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This  report  describes  the  progress  to  date  of  an  effort  to  develop 
new  concepts  of  management  applicable  to  tlie  U.S.  Bureau  of  Mines' 
program  of  Energ>'  Researcli.    A  systems  analysis  task  c^oup  was  organized 
to  identify  principal  problems  associated  with  implementing  a  mission- 
oriented  research  program    and,  in  particular,  to  design  a  mechanism  for 
detennining  researcli  project  priorities  in  terms  of  stated  goals. 

The  Phase  I  Exploratory  Survey,  consisting  of  14  group  interviews  at 
the  Washington  Headquarters  and  at  two  operational  Research  Centers,  is 
detailed.    Results  thus  far  include  listings  of  acknowledged  "problem 
areas,"  uiiere  reasonable  improvements  are  considered  possible.    In  addition, 
a  metiiodology  analogous  to  the  "peer-review"  procedure  used  elsewhere  in 
government  for  evaluating  research  projects  in  terms  of  "technical  quality" 
and  "program  re. evance"  was  introduced  and  tested  m  a  simulation  exercise. 
Preliminary  results  were  encouraging  and  suggest  direction  for  the  additional 
work  needed  before  this  mechanism  can  become  a  useful  management  tool  for 
budget  review  purposes.    A  second  exercise  is  recommended  involvirig  all  six 
USBM  Energy  Resarch  Centers  in  conjunction  witJi  a  forthcoming  Mid-Year  Budget 
Review. 
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TOVJARD  NEW  CONCEPTS  OF  MANAGEMENT  APPLICABLE  TO  THE  U.S.  BUREAU  OF  MINES' 

ENERGY  RESEARCH  PROGRAM 


1 .  INTRODUCTION 

This  report  describes  the  first  three  months  of  work  performed 
under  an  Interagency  Agreement  by  which  the  National  Bureau  of  Standards 
(Department  of  Commerce)  is  to  assist  the  Energy  Research  Group  of  the 
Bureau  of  Mines  (Department  of  Interior)  in  developing  new  concepts  of 
Management  applicable  to  their  program  of  Energy  Research. 

This  work  comprised  the  initial  part  of  an  Exploratory  Survey  aimed  at 
clarifying  principal  issues  concerning  the  newly  integrated  program  of 
Energy  Research  within  the  Bureau  of  Mines' research  organization.  Such 
exploration  and  clarification  necessarily  precede  the  formulation  of 
possible  means  for  improvement.    Specifically,  the  task  involved  the 
identification  of  actual  problems  associated  with  implementing  mission- 
oriented  research  projects  within  the  context  of  the  existing  six  Energy 
Research  Centers  and  Laboratories  operated  by  the  Bureau  of  Mines.  To 
date,  only  two  of  tlie  six  Research  Centers  have  been  visited. 

To  perform  this  survey,  a  joint  task  group  x^^as  organized  by  the 
Technical  Analysis  Division  of  NBS  and  requested  to  make  an  objective 
examination  of  the  functional  operating  activities  and  principal  informa- 
tion flows  at  three  decision  levels  within  the  Bureau  of  Mines  Energy 
Research  organization.    The  Task  Group  (called  "Project  SOAP"*)  conducted 

^Project  SOAP:    A  systems  analysis  task  group  identified  by  the  acronym 
for  ^Systems/Operation  Analysis  of  programs.    The  term  "systems  analysis" 
refers  to  a  method  for  determining  information  relevant  to  managing  an 
organized  activity,  for  the  purpose  of  improving  its  decision -making  pro- 
cess in  a  more  rational  way.  In  the  context  of  this  assignment,  it  is 
intended  to  be  used  as  a  prerequisite  for  inplementing  the  discipline  of 
Program  Management.  i 


a  series  of  14  group  interviews  in  April  and  May  1973  during  visits 
to  the  Washington  Headquarters  and  to  two  of  the  six  field  locations,  the 
Morgantown  Energy  Research  Center  (MERC)  in  West  Virginia  and  the  Bartles- 
ville  Energy  Research  Center  (BERC)  in  Oklahoma. 

This  status  report  summarizes  the  work  activities  of  the  Project 
SOAP  task  group  from  March  26th,  through  June  30,  1973,  as  required  by  the 
terms  of  the  Interagency  Working  Fund  Agreement*.    It  describes  the  type 
of  approach  employed  and  identifies  the  progress  made,  to  date,  as  well  as 
some  of  the  difficulties  encountered  during  this  attempt  to  improve  the 
general  understanding  of  the  decision-making  process  for  managing  the  Energy 
Research  activities  of  the  Bureau  of  Mines. 

The    details  of  the  initial  part  of  this  Exploratory  Survey  (covering 
field  visits  to  the  Washington  Headquarters,  MERC  and  BERC  only)  are 
presented  in  Section  2  of  this  report.    Preliminary  findings  from  the  two 
visits  to  MERC  (during  April  1973)  are  presented  in  Section  3  and  were 
reported  at    a    briefing  to  the  Washington  Staff  and  Research  Directors, 
Energy  Research  Group,  at  their  annual  meeting  in  Washington,  May  1,  1973. 
Also,  an  informal  briefing  following  the  field  trip  to  BERC  was  given  to  the 
Project  Officer  and  the  Assistant  Director -Energy  on  May  30,  1973. 

In  order  of  presentation,  this  report  first  outlines  the  background 
for  this  assignment  and  identifies  its  scope  and  objectives;  then,  in  the 
main  body,  it  briefly  describes  the  two  field  visits  (to  MERC  and  BERC); 


*In  accordance  with  provisions  of  Public  Law  89-672  (80  Stat,  951); 
Public  Law  92-369  (86-Stat.  514);  and  Section  601  of  the  Federal 
Economy  Act,  APPVD  June  30,  1932,  as  amended  (31  USC  686). 
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and,  lastly,  in  Section  5,  it  summarizes  the  principal  findings  and 
conclusions  to  date.    No  formal  analyses  or  recommendations  are  included 
in  this  first  status  report.    The  ^pendix  contains  material  referenced 
in  the  main  text. 
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1.1    Background  Perspective 

In  his  first  two  Energy  Messages  to  Congress,*  the  President  established 
guidelines  for  the  development  by  the  Federal  government  of  a  National  Energy 
Policy  and  Program  to  assure  the  country  has  a  sufficient  supply  of 
energy  at  reasonable  costs,  while  providing  adequate  incentives  for 
Industry  to  sustain  a  healthy  economic  growth  and  improve  the  quality  of 
national  life. 

To  implement  this  Policy,  the  Secretary  of  Interior  accepted  the 
responsibility  for  establishing  an  Energy  and  Minerals  Mission,  and 
charged  the  U.S.  Bureau  of  Mines  (USBM)  with  a  portion  of  the  Interior 
Department's  mission  pertaining  to  Energy  Research. 

The  Energy  Research  Mission  of  the  U.S.  Bureau  of  Mines  was  recently 
formulated  with  the  following  primary  goal:** 

To  develop  effective  means  that  will  increase  the  availability 
and  improve  the  utilization  of  the  Nation's  energy  resources  at 
reasonable  costs  and  in  a  manner  that  will  help  attain  desirable 
social  and  environmental  improvements. 

To  meet  this  important  challenge,  the  existing  organizational  resources 
of  USBM  are  to  be  combined  into  an  integrated,  mission-oriented  Energy 
Research  Program.    Currently,    specific  areas  of  commodity  research 
are  conducted  separately,  and  somewhat  independently,  at  each  of  six 

*March  10,  1972,  and  April  18,  1973. 

*USBM  Energy  Research  Program  Mission  Statement,  l^y  25,  1972. 
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principal  USBM  Research  Centers  and  Laboratories*  employing  about  900 
people  across  the  United  States. 

From  the  Bureau  of  Mines'  Budget  of  FY  73,  almost  $20  million  was 
spent  on  research  and  development  allocated  to  the  four  separate  commodity 
research  areas:    coal,  petroleum,  natural  gas,  and  oil  shale.    In  order 
to  respond  more  effectively  to  the  R§D  priorities  in  the  President's 
Energy  Messages,  the  USBM  has  recently  initiated  a  new  concept  of  mission- 
oriented  program  planning  and  evaluation.    The  primary  aim  of  instituting 
such  explicit  management  methods  for  its  research  activities  is  to  assist 
the  six  Researcli  Center  Directors  in  developing  a  more  rational  and 
coordinated  approach  for  allocating  research  resources  toward  recognized 
national  Energy  goals,  and  to  provide  a  consistent  way  of  monitoring 
progress  toward  those  goals. 

To  help  organize  the  major  research  activities  (and  their  relative 
priorities)  to  be  undertaken  by  the  various  Centers  and  Laboratories, 
an  integrated  Energy  Research  Mission  Statement  was  prepared  in  May  of 
1972.    It  was  hoped  that  this  "Mission  Statement"  would  provide  tiie  basis 
for  planning  and  evaluating  the  various  USBM  research  projects  in  terms 
of  the  "Energy  Mission"  objectives.    This  document,  while  useful  at  the 
Washington  Headquarters  level,  was  not  immediately  accepted  at  the  field 
level  because  of  the  precedent**it  established.     (It  has  since  been  thoroughly 
reviewed,  however,  at  both  the  Washington  and  field  levels,  and  is  expected 
to  be  released  shortly.) 

*Morgantown  Energy  Research  Center  (MERC),  Morgantown,  West  Virginia; 
Pittsburgh  Energy  Research  Center  (PERC) ,  Pittsburgh,  Pennsylvania; 
Bartlesville  Energy  Research  Center  (BERC) ,  Bartlesville,  Oklahoma; 
Larmie  Energy  Research  Center  (LERC) ,  Laramie,  Wyoming;  San  Francisco 
Energy  Research  Laboratory,  San  Francisco,  California;  Grand  Forks 
Energy  Research  Laboratory,  Grand  Forks,  North  Dakota. 

^*That  of  an  integrated  research,  program  with  its  strategic  technological 
goals  detemined  by  the  Washington  Headquarters. 
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From  this  prolonged  review  of  the  Energy  Research  Mission  Statement 
document,  it  appeared  there  were  "communications  problems"  between  the 
Headquarters  Staff  and  the  Researchers  in  the  Field  concerning  Research 
priorities.    In  addition,  a  mechanisr..  for  implementing  the  new  management 
concepts  presented  in  this  Statement  was  needed. 

In  February  1973,  through  the  recommendation  of  Mr.  Harry  R.  Johnson, 
Assistant  to  the  Assistant  Director -Energy  (by  way  of  Mr.  Leo  A.  Schrider 
[^'1ERC]  and  Dr.  Jack  Byrd,  Professor  of  Industrial  Engineering  at  West 
Virginia  University) ,  a  report  entitled  "Toward  A  Discipline  for  Program 
Management  (A  Summary  of  Project  SOAP  Activities)"  came  to  the  attention 
of  the  Bureau  of  Mines.    This  report,  presented  at  the  1973  TIMS/ORSA  Meeting* 
in  Atlantic  City  on  November  7,  1972,  was  the  result  of  a  two-year  project 
conducted  by  a  task  group  from  the  Technical  Analysis  Division  of  the 
National  Bureau  of  Standards.    It  describes  an  application  of  Management 
Science  by  a  systems  analysis  task  group  to  resource  allocation  problems 
within  a  sister  Federal  research  agency- -the  National  Institute  of  Allergy 
and  Infectious  Diseases  (NIAID)  of  the  National  Institutes  of  Health.  Johnson 
and  Schrider  were  of  the  opinion  that  the  methods  used  by  the  SOAP  team  within 
NIAID  might  be  applicable  to  certain  Program  Management  problems  withiii  the 
USBM  Energy  groups. 

Accordingly,  in  early  fferch  1973,  preliminary  discussions  were  held 
between  members  of  the  Bureau  of  Mines  and  the  National  Bureau  of  Standards . 
On  March  26,  1973,  Interagency  Agreement  WFA-SO  133054  was  executed 
authorizing  the  initial  part  of  this  four-phased  assignment. 


Joint  Meeting  of  the  Operations  Research  Society  of  America    and  The 
Institute  of  Management  Sciences,  Atlantic  City,  N.J.,  November  7-9,  1972. 
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The  following  is  a  detailed  description  of  the  Task  Group's 
objectives,  its  approach,  and  findings  during  the  first  part  of  its  first 
pliase--the  Exploratory  Survey, 
1.2    Scope  of  Assignment 

Tliis  Interagency  Agreement  between  the  U.S.  Bureau  of  Mines,  Depart- 
ment of  Interior,  and  the  National  Bureau  of  Standards,  Department  of 
Commerce,  was  undertaken  to  assist  the  Bureau  of  Mines  Energy  Group  develop 
new  concepts  of  management  applicable  to  tlieir  program  of  Energy  Research. 
(The  complete  Assignment  consists  of  four  piiases:    Phase  I  -  Exploratory; 
Phase  II  -  Analysis;  Phase  III  -  Prescriptive;  and  Phase  IV  -  Implementation.) 

The  primary  purpose  of  the    initial  Phase  I  Exploratory  effort  was 
to  seek  the  clarification  of  principal  issues  associated  with  the  imple- 
mentation of  mission-oriented  Program  Management  concepts  within  the  USBM 
Energy  Group. 

Tlie  scope  of  the  agreement  was  limited  to  only  the  first  part  of 
tliat  phase:    to  visits  to  the  Washington  Headquarters  and  to  two'^ 
operational  Field  Centers  (Morgantown,  West  Virginia,  and  Bartlesville, 
Oklalioma)  ,    Emphasis  was  directed  toward  investigating  tlie  potential 
usefulness  of  an  explicit  mechanism  for  evaluating  research  projects. 
It  was  also  expected  that  this  survey  would  provide  the  organization  with 
the  broader  perspective  required  to  coordinate  some  of  the  Bureau  of 
Mines'  other  Program  Management  activities. 
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1.2.1  Objectives 

Specifically,  the  joint  ta^k  group*  was  organized  to  perform  the 
following  tasks: 

•  To  improve  the  general  understanding  of*  the  External/ Internal  and 
Institutional  context  within  which  the  decision-making 

process  operates. 

•  To  clarify  principal  issues  and  identify  "tractable 
problems"  associated  with  the  implementation  of  mission- 
oriented  R§D  projects.  '^^  ' 

■     •    To  define  a  mechanism  for  evaluating  "Quality"  and 
"Relevance"  of  research  projects  in  terms  of  stated 
program  goals.  -      •  •      ' '■■  ' 

1.2.2  Approach  '  .  ^       r    .s  •  -  ■; 

The  method  the  SOAP  team  used  was  the  surveying  of  various  peer  groups 
within  each  location- -at  ^4BRC,  BERC,  and  the  Washington  Headquarters --to 
determine  the  existing  functional  information  flows,  feedbacks,  and 
decision  points  in  order  to  identify  the  nature  and  extent  (and  precise 
location)  of  significant  problems  amenable  to  solution  by  improved  manage- 
ment methods.    Tlie  mode  of  operation  was  the  face-to-face,  open-ended  group 
interview  technique,  with  the  above  three-point  agenda  used  for  focusing 
discussion  on  the  objectives  of  the  task  group. 

*The  Project  SOAP  task  group  members  included:    Robert  S.  Cutler, 
Vincent  A.  Martino,  and  Dr.  Mary  Ellen  Golby  from  TAD/NBS,  and 
Dr.  Jerry  D,  Ham  and  Harry  R.  Johnson  (Project  Officer)  from  USBM. 
(At  MERC,  the  group  was  assisted  by  Leo  A.  Schrider,  and  at  BERC, 
by  William  D.  Howell.) 
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1.2.3    Preliminary  Results 

The  results  anticipated  from  this  survey  included  a  briefing  delivered 

on  May  1  and  this  report,  describing: 

•  Details  of  work  performed,  to  date,  under  the  Interagency 
Agreement . 

•  Listing  of  acknowledged  "Problem  Areas"  where  reasonable 
improvements  are  considered  possible. 

•  A  recommendation  concerning  whether  the  use  of  quantitative 
methods  for  implementing  new  concepts  of  program  management 
within  USBM  Energy  Research  Centers  and  Laboratories  is 
operationally  feasible. 

To  date,  the  Project  SOAP  task  group  has  followed  an  orderly  procedure 
for  establishing  a  firm  factual  basis  concerning  the  nature  of  actual 
"problem  areas"  identified  at  each  operating  location  visited.    The  pre- 
liminary results  obtained  thus  far  include  statements  of  acknowledged 
"problem  areas"  considered  amenable  to  solution  by  available  management 
methods.    These  results  are  contained  in  Section  3  of  this  report. 

It  would  be  unreasonable  to  expect  that  any  task  group  could  attain 
such  insight  into  the  myriad  problems  of  a  complex  organization  after  just 
two  site  visits  as  to  identify  any  single  problem  as  the  crux  of  the 
matter  and  recommend  how  it  should  be  solved.    Only  after  completing  its 
exploratory  work,  which  assures  that  the  factual  foundation  is  properly 
laid,  can  the  SOAP  team  engage  in  a  more  compreKenfive  analysis  of  the 
principal  problems  associated  with  this  assignment- -the  development  of  a 
new  concept  of  management  applicable  to  the  entire  USBM  Energy  Research 
Program . 
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From  its  findings  thus  far,  from  the  14  interviews  with  groups  at 
three  locations  within  the  USBM  Energy  Research  organization  and  from  the 
preliminary  results  of  Exercise  I*,  the  Project  SOAP  task  group  concludes 
that  it  is  worthwhile  to  proceed  with  the  additional  work  needed  to 
implement  an  operational  "peer-review"  mechanism  for  evaluating  individual 
research  projects  in  terms  of  program  goals.  ■ 


*A  preliminary  assessment  of  a  sample  set  of  Research  Projects  involving 
56  individuals  in  six  groups  at  three  locations  (WASH/MERC/ BERC)  during 
May  1973.    See  Subsections  2.4  and  3.3  for  details. 


10 


2.    PHASE  I:    EXPLORATORY  SURVEY 


The  following  details  the  activities  performed  and  the  observations 
made  by  the  Project  SOAP  team  during  the  initial  part  of  this  Exploratory 
Survey. 

2.0.1    Project  SOAP  Orientation 

A  preliminary  meeting  was  held  March  7,  1973,  at  the  National  Bureau 
of  Standards  between  three  members  of  the  U.  S.  Bureau  of  Mines  Program  of 
Energy  Research  and  five  members  of  the  Technical  Analysis  Division  to 
discuss  more  fully  the  possibility  of  TAD's  people  doing  a  systems  analysis 
study  similar  to  the  one  previously  conducted  by  TAD  for  one  of  the 
National  Institutes  of  Health.       It  was  to  be,  essentially,  a  study  of 
Program  Management  conditions  within  the  Bureau  of  Mines'  Energy  Research 
Centers . 

After  a  lengthy  discussion  devoted  to  the  managerial  structure 
of  the  USBM  Energy  Program,  the  USBM  people  stressed  that  the  main  problem 
of  their  Program,  as  they  understood  it,  was  one  of  communication  and  the 
flow  of  technical  ideas  and  information- -of  vertical  communication  between 
Headquarters  and  the  Field,  and  of  horizontal  communication  between 
various  levels  of  one  Research  Center  and  those  of  the  other  Centers. 
Somewhere- -they  didn't  know  precisely  where- -there  were  barriers  to  this 
vital  communications  flow. 


From  USBM:    Harry  R.  Johnson,  Dr.  Andrew  W.  Decora,  Leo  A.  Schrider; 
from  TAD:    Louis  C.  Santone,  Robert  S.  Cutler,  Vincent  A.  Martino, 
David  M.  Clancy,  Dr.  Mary  Ellen  Colby. 

**Cutler,  Robert  S.,  "Project  SOAP:  A^^ystem,s  Approach  to  Biomedical  Research 
Program  Management -A  Case  Study,"  NBS  Technical  Note  761,  CApril  197^). 
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The  TAD  people  noted  that,  in  addition  to  the  "communications"  problem, 
there  also  seemed  to  be  a  "control"  problem  and  that,  before  considering 
any  new  management  concepts,  they  ought  to  understand  the  nature  of  the 
current  resource  allocation  process  used  by  USBM  throughout  its  Energy 
Research  Program.    However,  as  was  clearly  pointed  out,  there  was  too 
little  time  to  deal  effectively  with  the  whole  organization.    It  was  finally 
agreed  that  an  initial  survey  covering  operations  at  the  Washington 
Headquarters  and  at  one  or  two  Field  Centers  would  constitute  the  initial 
part  of  this  Exploratory  Phase. 

The  next  step  recommended  by  the  TAD/ SOAP  team  members  was  that  they 
be  permitted  to  meet  as  many  of  the  USBM  people  knowledgeable  about 
USBM  Energy  Research  as  possible  before  visiting  the  Field  Stations,  because 
the  team  operated  then,  as  always,  upon  the  premise  that  problem- finding 
must  necessarily  precede  problem- solving.    Dr.  Jerry  Ham  in  the  Arlington 
office  and  Mr.  Harry  Johnson  in  the  Washington  office  volunteered  to  help 
set  up  meetings  with  people  they  felt  the  SOAP  team  should  interview, 
including  Mr.  William  L.  Crentz,  the  Assistant  Director  of  the  USBM  program 
of  Energy  Research;  Dr.  G.  Alex  Mills,  Chief  of  the  Division  of  Coal; 
J.  Wade  Watkins,  Chief  of  the  Division  of  Petroleum  and  Natural  Gas;  and 
Jack  Phillips,  Chief  of  Shale  Oil,  and  their  immediate  Staffs. 
2.0.2    The  Initial  Task        .    .....  , 

Initially  there  was  an  "identity  problem"  to  solve.    Because  the 
aim  of  this  assignment  and  the  objectives  of  this  survey  were  not  clearly 
understood  by  everyone  in  advance,  the  USBM  Energy  Program  personnel 
felt  uncertain  about  what  to  expect  in  terms  of  project  results.    Part  of  this 


*See  letter,  Santone,  L.  C.  to  Crentz,  W.  L.,  dated  March  9,  1973  in 
Appendix,  pp.  A- 2. 
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difficulty  lay  in  their  skepticism  about  the  usefulness  of  system  analysis 
techniques.    Similarly,  the  systems  analysis  people  from  TAD/NBS  recognized 
their  lack  of  familiarity  with  the  technology  of  Energy  Research.  Thus, 
there  were  mutually  recognized  difficulties  in  achieving  desired 
results  within  the  allotted  time.    Fortunately,  however,  there  was  also 
a  sincere  resolve  on  both  sides  to  make  a  meaningful  and  productive  start. 
Dr.  Jerry  D.  Ham,  Petroleum  Engineer,  USBM  Washington  Staff,  was  assigned 
part-time  to  the  Project  SOAP  team  to  act  as  the  liaison  with  the  operating 
units  of  the  USBM  Energy  Group. 
2.1   Washington  Staff  Interviews 

A  series  of  meetings  was  scheduled  with  members  of  the  Washington 
Staff  of  the  USBM  Energy  Research  group  to  acquaint  them  with  the  purpose 
of  Project  SOAP  and  gain  their  guidance  on  things  to  look  for  during  the 
field  trips.    In  these  (as  in  later  interviews)  the  SOAP  team  was  cordially 
received,  even  when  healthy  skepticism  was  voiced  about  what  the  team  was 
supposed  to  do. 

Following  is  a  list  of  the  initial  Washington  Staff  meetings  held 
during  this  time: 

j'^ril  2.    Orientation  Meeting  with  Dr.  Jerry  Ham  (USBM/Arlington) , 
at  NBS. 

April  3.    Initial  Meeting  with  Mr.  Crentz,  Asst.  Director -Energy 
(USBM),  in  his  office,  in  Washington,  D.C. 

April  3.    Orientation  Meeting  with  Messrs.  Watkins,  Phillips, 
Gooding,  Ramsey,  and  Dr.  Ham,  in  their  office  at 
Arlington,  Va. 
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April  4.    Orientation  Meetings  with  Dr.  Mills  and  with  Mr.  Corey 
,       (USBM)  in  their  offices,  in  Washington,  D.C. 

As  a  result  of  these  initial  discussions,  the  SOAP  team  gathered  the 
following  information  and  impressions  to  guide  its  further  investigations: 
(1)  There  is  no  mission  "indoctrination"  for  new  researchers 

•  at  the  various  Centers,  either  initially  or  subsequently. 
C2}  Management  pressures  (e.g.,  influence  of  labor  unions)  are 

extremely  important  at  some  Centers,  almost  non-existent  at  others 

(3)  There  tends  to  be  too  much  parochialism  (single -commodity 
interest)  at  all  of  the  Centers. 

(4)  The  Research  Supervisors  appear  to  be  the  most  inflexible 
link  in  the  Centers'  managerial  chain- of- command. 

(5)  The  general  career  history  of  the  Centers'  research  people 
is  one  of  migration  from  universities  into  business  and  then 
to  government. 

(6)  The  presence  of  a  university  near  a  Research  Center  tends  to 
intellectually  stimulate  the  atmosphere  of  the  Center. 

(7)  Washington  tends  to  lose  sight  of  changing  technologies, 

•  while  people  at  the  Centers  do  not. 

(8)  Washington  feels  the  need  for  a  study  of  the  "relevance"  of 
the  research  conducted  at  the  USBM  Energy  Research  Centers. 

(9)  Researchers  in  the  Field  tend  to  have  no  concept  of  completion 
time- frame. 

(10)  If  the  new  Mid-Year  Review  is  to  be  useful  to  people  other  than 
just  the  Asst.  Director-Energy,  review  comments  should  always  be 
followed  by  a  feedback  to  the  field  ^s  was  recently  done  for 
FY73) . 


(11)  The  Bureau  of  Mines  seems  to  have  more  difficulty  selling 
its  programs  to  the  Department  of  Interior  and  0MB 

than  to  Congress. 

(12)  Many  line-decisions  (directives)  formerly  made  in  the  Field 
are  now  made  by  the  Washington  Staff. 

(13)  There  is  a  mutual  Headquarters/Field  need  for  understanding 
the  functional  decision-making  processes  at  the  Centers. 

(14)  There  is  something  believed  wrong  with  the  Bureau  of  Mines' 
present  resource  allocation  methods. 

(15)  Something  is  needed  to  improve  the  effectiveness  of  the  "New 
Idea  Referral  System." 

It  was  also  suggested,  later,  by  Mr.  Crentz  that  the  SOAP  team 
visit  not  MERC  and  PERC,  as  originally  planned  (they  had  the  virtue  of 
proximity),  but  MERC  and  BERC,*  by  way  of  contrast.    Located  in  different 
areas  of  the  country- -West  Virginia  and  Oklahoma- -they  had  unique 
environments  and  might  perhaps  exhibit  contrasting  problems. 
2.2    Morgantown  Energy  Research  Center  Interviews 

Before  the  SOAP  team  arrived  at  the  Morgantown  Center  they  were 
aware  of  the  existence  of  certain  stresses  there.    For  one  thing, 
there  had  been  a  Reduction  in  Force  (RIF)  the  last  fiscal  year  and  the 
prospect  of  another  one  during  the  next  fiscal  year,  a  situation 
which  did  not  help  morale.    For  another,  there  was  an  organized  Labor 
Union  Negotiation  pending  at  the  time.    Then,  too,,  the  Center's  former 


For  acronyms,  see  p.  5. 
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Director  had  recently  died  and  his  successor,  Mr.  William  C.  Eckard, 
had  barely  arrived.  Lastly,  there  was  the  imminent  possibility  that 
the  proposed  FY74  budget  of  $3.5  million  was  going  to  be  cut  to  $2.7 
million.  . 

Fortunately,  however,  the  new  Center  Director  had  taken  positive 
steps  to  understand  the  precise  nature  o£  the  problems  which  currently 
existed  at  MERC.    He  had  created  several  internal  task  forces  from  among 
his  people,  thirteen  of  whom  had  recently  participated  in  a  Leadership 
Systems  Workshop.*    Because  of  these  internal  task  force  efforts,  the 
SOAP  team  had  an  easier  time  identifying  significant  "problem  areas" 
than  it  might  otherwise  have  had.    For  example,  one  MERC  task  force  had 
concluded  that  employees  have  "great  difficulty. .. in  getting  a  fair 
and  reasonable  hearing  for  their  ideas  on  possible  new  projects  that  fit... 
[their]  mission."       They  stated  that  the  "idea  referral  system  is  pitiful" 
and  that  the  "project  proposal  system  is  fine  in  theory,  but  poor  in 
practice.    The  major  problem  is  not  the  mechanics  of  these  systems,  but 
rather  [the]  failure  of  the  supervisory  chain  to  nurture  the  system  and 
make  it  work."    They  complained  that  "little  information  goes  below  the 
Research  Supervisor  level,"  including  mission  statements  and  budget  infor- 
mation, and  they  called  for  more  visits  from  the  Washington  Staff  to  MERC.*** 

^  ^  1^  ^ 

All  seemed  to  agree  that  "communications  was  a  part  of  all  the  problems," 


_^*Sponsored  by  the  Bureau  of  Mines. 

"Report  of  Task  Force  to  Consider  Recommendation  to  Maximize  Productivity 
and  Job  Satisfaction  in  MRED  and  MERC,"  February  14,  1973,  p.  2. 
***Ibid.,  p.  3. 

****"Report  of  Task  Force  on  Communication  and  Organization,"  March  19,  1973, 
Introduction. 


16 


though  they  believed  (contrary  to  what  SOAP  was  to  learn  that  Washington 
Headquarters  actually  wants)  that  the  job  of  the  Research  Supervisor  was 
to  "sell  the  Washington  staff  on...  [their]  i^e.as  for  new  or  continuing 
programs."     They  called  for  two  Mission  Statements,  one  from  Washington 
and  one  from  MERC,  wherein  guidelines  "should  be  developed  to  outline  areas 
in  which  project  proposals  are  described  within  the  framework  of  both 
mission  statements."       They  also  called  for  the  selection  of  Project 
Managers  from  within  the  Center,***  in  cases  where  extremely  important  pro- 
jects were  involved. 

It  was  evident  from  all  this  that  MERC  not  only  had  serious  problems 
but  was  keenly  aware  of  them,    ivforeover,  it  was  attempting,  largely  as  a 
result  of  the  stimulus  of  their  new  Director,  to  do  something  about  them. 
2.2.1    MERC  Visit  #1  (April  10-11,  1975) 

The  first  group  interviewed  by  the  Project  SOAP  team        the  afternoon 
of  April  10  was  comprised  of  the  Research  Center  Director,  the  Deputy 
Director,  and  the  Center's  five  Research  Supervisors.    These  men,  in  turn, 
were  later  asked  to  select  three  men  from  each  of  their  sections  to  attend 
a  similar  meeting  to  be  held  the  following  day.    Mr.  Leo  A.  Schrider, 
Petroleum  Engineer,  was  asked  to  assist  the  SOAP  team  by  providing 
necessary  support  to  the  group  while  at  MERC. 

In  an  attempt  to  make  proper  introductions  and  to  dispel  suspicion 
that  the  Project  SOAP  team  had  been  sent  down  by  top  management  in 
Washington  to  conduct  a  "witch-hunt,"  the  Director,  Mr.  William  E.  Eckard, 


* 


Ibid. ,  p.  3. 
Ibid. ,  p.  4. 
Ibid. ,  p.  6. 


**** 


Robert  S.  Cutler,  Vincent  A.  Martino,  David  M.  Clancy,  Dr.  Mary  Ellen  Colby 
(all  of  TAD) ,  and  Dr.  Jerry  Ham  (USBM,  Arlington) . 
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introduced  the  individual  members  o£  the  SOAP  team  before  each  meeting. 
He  explained  their  mission  and  how  they  came  to  be  involved  in  it,  and 
then  he  discreetly  disappeared  (except  for  the  meeting  with  the  Research 
Supervisors)  in  order  to  promote  complete  frankness  on  the  part  of  the  others. 
The  team  learned  that  there  was  a  general  feeling  within  the  Center  that 
nobody  up  there  (in  Washington)  cared  about  them- -a  feeling  the  team 
attempted  to  probe,  pointing  out  that  their  mere  presence  there  to  help 
them  identify  "problem  areas"  was  proof  that  "someone  up  there"  did,  indeed, 
care. 

Early  in  the  course  of  these  meetings,  and  quite  by  accident,  the 
term  "Research  Monitor"  came  up.    When  the  men  at  MERC  were  asked  to 
explain  who  this  person  is  and  what  he  was  supposed  to  do  (i.e.,  what 
his  functions  was),  they  were  unable  to  be  precise. 

It  was  then  that  Dr.  Ham  mentioned  that  he  was  a  Research  Monitor; 
but  he,  too,  could  not  give  a  definitive  answer  on  the  subject.  (Nor 
is  this  surprising,  because,  as  will  be  seen,  there  is  great  confusion  among 
the  Washington  Staff- -among  the  Research  Monitors  themselves- -about  the 
exact  function  of  this  person.)    The  SOAP  team  then  concluded  that  one 
of  the  first  things  it  would  have  to  do  once  back  in  Washington  was 
find  out  more  about  the  role  of  the  Research  Monitor.    He  loomed  too  large 
on  the  horizon  to  be  overlooked;  moreover,  the  uncertainty  as  to  his 
function  seemed  symptomatic  of  the  general  lack  of  communications 
between  the  Field  and  Headquarters .  '  — 
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Back  in  Washington,  the  team  reviewed  their  notes  and  impressions 
from  the  two  previous  days  of  meetings  and  developed  a  tentative  listing 
of  "identified  problem  areas"  (shown  below)  believed  amenable  to  solution. 
The  next  step  was  to  report  to  the  Assistant  Director  what  they  had 
learned,  at  MERC,  about  the  confusion  over  the  functions  of  the 
"Research  Nfonitor."    They  then  asked  for  permission  to  schedule  a  return 
trip  to  Morgantown  in  order  to  ascertain  whether  the     identified  problems 
would  in  fact  be  acknowledged  as  real  and  salient  by  the  participants  in 
the  initial  discussions  at  MERC. 

The  following  lists  the  findings  from  this  first  visit: 

"Problem  Areas"  Identified  frra  Init-al  Visit  To  >Torgant(^>-n 
Enerp'  Rcsearcli  Center,  .■\pril  lQ-11,  1971 

1.  Need  for  r.xplicit  Station  Mission  Statcmerit. 

•  Translation  of  Energy  Research  Progran  "Goals" 
into  Research  Project  "Cbjectives." 

•  Participation  in  Project -Planning  Process. 

•  Results -Oriented  Task  Group  Approach. 

2.  Clarif)-  Research  Nbnitor  Function. 

•  Explicit  Role  in  "Cammi  cat  ions"  Process. 

•  Strategic-Level  Plaming  and  Assessments. 

•  Program  Planning/Evaluation  Staff. 

3.  Inproved  Mechanism  for  Evaluating  Projects  in  Terms  of 
Stated  Progi-an  Coals . 

I   Project  Priority  Determination. 

•  "Quality"  (Teer-F^view  Judgements) . 

•  "Relevance"  (Inten.al/Extemal  Opinions) . 
»   Time-Frame  for  P^sults. 

•  Costs  O-Kalti-Year ,  Total  Investment). 

4.  Iniproved  Organizational  "Feedback." 

•  Intra-Communications  0^'ithin  Center)  . 

•  Inter-Ccnnuriications  CBetween  Centers,  HQTRS/Field) . 

•  Participation  in  Decision  Process. 

5.  Idea  Development/Project  Proposal  System. 

•  Stirulate  "Good"  P^search  Projects. 

•  Assure  Useful  Peseardi  Results. 

•  Consistent  Criteria. 
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On  April  16  and  18,  before  returning  to  MERC,  the  SOAP  team  met 
all  of  the  ten  Washington  Research  Monitors  who  were  in  town  and  who  did 
not  have  prior  commitments.     The  gist  of  what  they  learned  from  these 
meetings  is  that  there  was  as  much  confusion  among  the  Research  Monitors 
themselves  concerning  their  role  as  there  was  among  the  men  at  MERC. 

While  the  men  at  Morgantovni  complained  of  the  lack  of  efficacious 
monitoring  of  programs  by  Washington  and  called  for  more  on-site 
visits  by  Monitors  (hopefully,  once  a  month),  SOAP  received  the  following 
diversity  of  comments  from  the  Research  Monitors  themselves:    (a)  that 
there  really  was  no  such  a  person  as  a  Research  Monitor;  (b)  that  there  was, 
indeed,  such  a  person,  and  he  was  very  valuable  to  the  decision-making  process; 
(c)  that  the  Monitors,  who  are  scientific  experts,  make  no  scientific 
judgements  on  the  scientific  proposals  that  are  sent  to  them;  (d)  that  the 
present  Research  Monitor  function  has  replaced  the  former  Program  Manager 
function- -and  the  latter  never  really  got  off  the  ground,  anyway;  (e)  that 
one  of  the  Research  Monitor's  functions,  in  addition  to  keeping  tabs  on 
the  research  going  on  in  the  Field,  is  to  keep  abreast  of  the  state-of- 
the-arts  in  industry  and  elsewhere;  (f)  that  all  of  the  Washington  Staff 
serve  as  Research  Monitors,  whether  or  not  officially  designated  as  such, 
and  that  frequently  one  substitutes  for  another;  (g)  that  the  actual 
function  of  the  Monitor  is  to  make  recommendations  to  the  Assistant  Director- 
Energy;  (h)  that  Research  Monitors  and  Research  Coordinators  are  one  and 
the  same;  (i)  that  at  least  one  Monitor  was  given  a  list  of  seven  projects 
he  had  never  seen  before  and  was  asked  to  give  his  opinion  on  them  within  48 
hours. 


Dr.  Jerry  Ham,  Richard  Gooding,  Larry  Burman,  Jerry  Ramsey,  Richard  Corey, 
Jack  Smith,  and  Brian  Harney. 

20 


2.2.2    MERC  Visit  #2  (April  25-26,  1975) 

On  April  25,  two  of  the  SOAP  team  members*  returned  to  the  Morgantown 
Energy  Research  Center.    The  visit  coincided  with  the  thirty- first  meeting 
of  the  Petroleum  Industry  Technical  Advisory  Committee  to  the  Morgantom 
Energy  Research  Center  on  Fluid  Energy  Resources.    The  SOAP  team  was 
invited  to  attend  as  observers.    Being  somewhat  ignorant  about  technical 
matters  up  to  now,  the  NBS  members  were  enlightened  by  the  proposals 
for  GAS  DEVEL,  Underground  Coal  Gasification  and  reports  of  Hydraulic 
Fracturing  and  Microseismic  Monitoring. 

The  second  day  of  this  visit  was  devoted  to  scheduled  meetings  with 
the  same  four  groups  of  the  MERC  personnel  the  team  had  met  with  on 
April  10-11.    After  further  discussion  and  feedback  of  what  the  team 
had  learned  in  Washington,  they  asked  each  of  the  four  groups  to 
re-rank,  in  order  of  their  concept  of  priority,  the  list  of  the  five 
identified  "problem  areas"  the  team  had  gleaned  from  its  initial  visit. 
The  results  of  these  rankings  are  show_  on  p.  31,  Section  3,  of  this 
report.    The  visit  concluded  with  a  brief  informal  report  to  the  Center 
Director,  Mr.  William  E.  Eckard. 


Robert  S.  Cutler  and  Dr.  Mary  Ellen  Colby.  (Harry  R.  Johnson,  from 
the  USBM  Washington  office,  was  also  present  for  the  conference.) 
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2.3    Bartlesville  Energy  Research  Center  Interviews 

Prior  to  its  visit  to  the  Bartlesville  Energy  Research  Center, 
the  SOAP  team  knew  little  about  either  the  Center  or  the  town.    It  had, 
however,  been  informed  that  Bartlesville  was  an  "oil"  community  (''It's  a 
one-company  town,"  the  team  was  told);  that  there  was  no  labor  union 
at  the  Center;  that  the  University  closest  to  the  Center,  Tulsa,  was 
sixty  miles  away;  that  one  of  the  best  Thermodynamics  groups  in  the  country 
was  employed  by  the  Center. 
2.3.1    BERC  Visit  #1  (May  21-22,  1973) 

On  Monday,  May  21,  1973,  two  SOP^  team  members*  met  with 
Mr.  John  S,  Ball,  the  Research  Director  of  BERC,  for  lunch.    During  this 
informal  gathering,  the  team  briefed  him  on  what  it  hoped  to  accomplish 
while  at  Bartlesville,  and  he,  in  turn,  briefed  the  team  on  the  situation 
at  BERC,  as  he  saw  it. 

The  SOAP  team  had  decided  well  in  advance  that  they  would  follow 
essentially  the  same  procedure  for  interviews  that  they  had  followed  at 
MERC.    They  requested  the  Center  Director  schedule  his  Research  Supervisors 
for  the  first  meeting  with  the  SOAP  team  and  that  the  Supervisors,  in  turn, 
nominate  additional  men  for  a  series  of  three  meetings  the  following  day. 
All  these  requests  were  granted. 

That  afternoon,  from  1:00-2:30  P.M.,  the  SOAP  team  met  with  the 
Director,  four  Research  Supervisors,  the  Superintendent,  and  the  appointed 
liaison  man,  William  D.  Howell  (who  attended  all  the  meetings) . 


^Robert  S.  Cutler  and  Vincent  A.  Martino,  of  TAD. 
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This  session  was  particularly  interesting  in  one  respect.  Although 
the  Director  introduced  the  SOAP  team  by  way  of  remarking  that  they  had 
not  been  sent  down  by  Headquarters  on  a  subversive  mission  but  had,  rather, 
come  to  identify  and  help  solve  problems,  there  existed  an  unmistakable 
air  of  skepticism.    Fully  two-thirds  of  this  meeting  was  spent  by  the 
SOAP  team  in  justifying  both  its  credentials  and  the  reason  for  its 
mission.    One  Si^ervisor  bluntly  challenged  the  assumption  that  there 
were  any  "problems"  at  BERC* 

However,  in  due  time,  these  "problems,"  voiced  as  complaints,  began 
to  surface  (and  were  duly  noted  by  the  SOAP  team) .  At  the  conclusion  of 
the  meeting,  the  Staff  agreed  to  select  three  men  each  from  their  groups 
to  attend  the  next  day's  sessions. 

The  three  groups  met  with  the  SOAP  team  on  May  22.    Again,  the 
Director,  John  S.  Ball,  introduced  the  SOAP  team  members  to  all  three  groups 
and  then  left,  in  an  attempt  to  assure  frankness  on  the  part  of  those 
participating.    And  the  men  were  frank.    That  night,  the  SOAP  team  drew 
up  the  following  list  of  "problem  areas"  they  had  identified  as  a  result 
of  these  four  meetings. 


*This  remark  was  all  the  more  interesting  in  view  of  the  following:  As 
at  MERC,  BERC  had  its  own  Leadership  Task  Force  on  improving  productivity 
and  job  satisfaction.    While  the  group  found  that,  on  the  plus  side, 
salaries,  facilities,  and  si^port  functions  were  good  at  Bartlesville, 
they  called  for  more  "active,  attentive  management  at  the  Washington  level"; 
the  creation  of  a  staff  to  serve  as  a  liaison  between  Washington  and  the 
field,  spending  much  of  its  time  in  the  field;  funding  of  only  that  research 
which  the  Bureau  can  best  perform;  better  communication,  both  vertical  and 
lateral;  and  "recognition  of  work  by  other  peer  groups."    See  Memorandum  on 
"Evaluation  of  Bureau  of  Mines  Management  Operation,"  Leadership  Systems,  Inc. 
Task  Force,  Bartlesville  Energy  Center,  March  7-12,  1973. 
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"Problem  Areas"  Identified  From  Initial  Visit  To 
Bartlesville  Energy  Research  Center,  May  21-23,  1975 

1.  Feelings  of  Frustration  Concerning  Inadequate  Funding  Supoort  for  BERC 
Research  Capabilities  in  Face  of  Current  National  Energy  Crisis 

•  Lack  Confidence  in  V/ashington  Salesmanship 

e      Need  for  Review  of  Policy  on  Outside  Agency  Funding 
e      Clarify  Budgetary  Decision  Process 

2.  Recognition  of  "Communication  Gap"  Between  Washington  Headquarters  and 
Field  Stations 

o      Clarification  of  Research  Direction  -  Mission  Statement 

•  Improved  Program  Planning/Evaluation  Staff  Capability 

«      Assignment  of  Soecific  Energy  Research  Program  "Requirements"  (and 
Priorities)  From  Headquarters  to  Centers 

3.  Clarification  of  Headquarters  Energy  Division  Staff  Function 
«      Research  Monitor/Coordination 

©      Headquarters- Level  Planning  and  Evaluation 

o      Liaison  with  Technical/industry  Advisory  Groups 

4.  Improved  Project  Management  Review  Mechanism 

A      Determination  cf  "Technical  Quality"  end  "Program  Relevance" 
of  Research  Activities 

•  Establish  Center  Priorities  - 
0      Provide  Basis  for  Program  Evaluation 

5.  New  Idea  Development/Profect  Proposal  System 

•  Present  Procedure  Too  Complicated,,  ineffective 

6      Lack  of  Incentive  (Funding)  to  Stimulate  innovation 

•  Require  Meaningful  Feedback  inilLn.^^f^^ 

RSC/YAM:V5C 
23  MAy  73 
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Because  they  had  only  the  morning  of  May  23rd.  for  quick  wrap-iq) 
sessions,  the  SOAP  team  alloted  approximately  one-half  hour  to  each  of 
four  meetings --with  the  Research  Siqjervisors  and  the  same  three  groups 
of  Project  Leaders  interviewed  the  preceding  day.    They  asked  the  Research 
Director  and  the  Research  Supervisors  to  review  and  re -rank  the  list 
of  five  identified  "problem  areas"  and  then  return  the  revised  listing  to 
TAD.    A  letter*  from  John  S.  Ball,  Research  Director  of  BERC,  describes 
the  results  of  the  process.    Instead  of  having  just  the  Research  Supervisors 
participate  in  Exercise  I,**  as  originally  planned,  the  team  requested  the 
participation  of  all  four  groups,  with  the  Research  Supervisors  and  one 
group  of  Project  Leaders  judging  "program  relevance"  only***  and  the  two 
other  groups  of  Project  Leaders  judging  "technical  quality"  only.  The 
men  were  asked  to  mail  their  tally  sheets  directly  to  TAD /MBS  by  June  1. 

The  next  step  for  the  SOAP  team  was  an  informal   feedback  report 
to  the  Washington  Staff  about  their  first  visit  to  BERC.    On  May  30,  1973, 
Mr.  Crentz,  Harry  Johnson,  Dr.  Ham  and  three  members  of  the  SOAP  team**** 
held  a  long  meeting  in  Mr.  Crentz 's  office.    Because  the  primary  aim  of  this 
session  was  the  recapitulation  of  the  team's  visit  to  Bartlesville,  the 
team  gave  a  detailed  account  of  their  stay  at  BERC,  including  a  contretemps 
between  one  of  their  members  and  one  of  BERC's  Project  Leaders,  with  apolo- 
gies from  the  involved  team  member. 


*See  letter,  John  S.  Ball,  Research  Director,  to  Robert  S.  Cutler, 
Project  SOAP,  dated  June  1,  1973  ,  in  Appendix,  pp.  A- 4  thru  A- 6 

**To  be  explained  more  fully  in  the  following  pages. 

***Because  certain  persons  in  one  group  were  not  qualified  to  judge  on 
technical  matters,  they  and  the  rest  of  their  group  were  asked  to  judge 
on  the  basis  of  their  concept  of  "program  relevance". 

****Robert  S.  Cutler,  Vincent  A.  Martino,  Dr.  Mary  Ellen  Golby,  all  of  TAD. 
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While  discussing  the  submitted  list  of  "problem  areas"  identified 
during  this  first  visit,  Mr.  Crentz  made  several  observations,  am.ong 
them  these  three,  which  are  very  pertinent  to  SOAP's  assessments  of  this 
Center's  relationship  to  its  Washington  Headquarters: 

•  Bartlesville  Energy  Research  Center  is  unique  in  its  funding. 
It  receives  approximately  50%  of  its  funding  from  agencies 
other  than  DOT --from  private  and  other  Government  agencies, 
as  well  as  from  the  state  of  Oklahoma.    Recently,  there  has 
been  a  steady  decrease  in  this  outside  funding,  and  BERC 

has  looked  to  USBM  to  fill  its  diminishing  coffer.    This  the 
USBM  has  been  unable  to  do,  simply  because  there  are  no  dollars 
to  be  had. 

•  To  the  charge  of  one  man  at  BERC- -that  one  of  the  reasons 
BERC  was  suffering  from  insufficient  funding  was  that  petro- 
leum research  did  not  have  the  powerful  lobby  on  the  Hill 
that  coal  research  had- -Mr.  Crentz  responded  negatively.  He 
pointed  out  that  the  petroleum  industry  has  no  difficulty  in 
making  its  voice  heard  in  Washington,  but  differs  considerably 
from  the  coal  industry  in  its  attitude  toward  and  reliance  on 
government  research.    Whereas  the  coal  industry  conducts 
virtually  no  research  of  its  own  (and  is,  therefore,  always 
delighted  when  the  government  offers  to  do  so) ,  petroleum  has 

a  multi -million  dollar  proprietary  research  program  and  tends  not 
to  seek  USBM  research  projects  precisely  because  they  are 
not  proprietary  and  tne  results  are  made  public  for  anyone  to 
exploit. 

•  The  most  fruitful  way  for  a  researcher  to  promulgate  a  new 
idea  is  to  get  the  Research  Director  behind  it.    If  he  confirms 
it  is  "good"  and  advances  the  idea  to  Washington  for  funding, 
Mr.  Crentz  believes,  "in  almost  no  cases  at  all,"  would  he  run 
into  a  stumbling  block.    The  logic  here  is  simply  that  if  the 
new  idea  is  actually  "good"  it  should  surely  be  sponsored  by 
the  Research  Supervisor  or  the  Research  Director,  and  it  will 
therefore  eventually  receive  funding  and  won't  be  the  victim 
of  the  funding  of  on-going  projects.    In  fact,  if  it  is 
sufficiently  good,  on-going  projects  may  be  its  victim.  (These 
remarks  were,  in  effects  ^  reiteration  of  Mr.  Crentz 's  state- 
ment to  the  Mid-Year  Review  Conference  for  May  1,  1973,  that 
he  wanted  two  sources  of  information  upon  which  to  base  his 
budgetary  decisions:    the  Research  Director  and  the  Division 
Chiefs. ) 
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2.4    Exercise  I:    Preliminary  Assessment  of  Research  Project  Proposals 

Following  its  May  1st.  briefing  to  the  USBM  Research  Directors,  who 
were  meeting  in  Washington  for  a  Mid-Year  Review,  the  SOAP  team  was  urged 
to  conduct  a  test  of  a  management  tool  for  evaluating  projects  in  terms 
of  stated  program  goals.    What  was  desired  was  a  means  for  ascertaining: 
(a)  how  consistently  various  people  associated  with  the  USBM  Program  of 
Energy  Research  would  rate  on-going  Project  Proposals,  and  (b)  what  their 
rationale  was. 

A  preliminary  Exercise  was  then  designed  to  determine  the  feasibility 
of  obtaining  a  meaningful  assessment  of  the  "quality"  and  "relevance"  of 
individual  research  projects  (and  their  criteria),  by  requesting  selected 
review  groups  to  evaluate  a  representative  sample  of  research  Project 
Proposals  submitted  to  the  U.S.  Bureau  of  Mines  for  FY- 74  funding. 

Two  separate  assessment  scores  were  sought:    one  for  "technical 
quality,"  and  the  other  for  "program  relevance."    These  scores  would  later* 
be  used  as  a  basis  for  comparing  the  resulting  rank  order  for  the 
sample  set  of  projects  with  the  actual  experience  of  the  organization  in 
establishing  priorities  for  the  funding  of  research  projects. 

Specifically,  the  objectives  of  this  Preliminary  Exercise  were: 

•  To  determine  the  feasibility  and  acceptability  of  evaluating 
Energy  Research  Project  Proposals    separately,  for  "technical 
quality"  and  "program  relevance." 

•  To  analyze  the  scoring  patterns  of  specific  review  groups 

in  determining  the  "relevance"  of  particular  Energy  Research 
projects  to  the  new  Energy  Research  mission  of  the  U.S. 
Bureau  of  Mines. 

•  To  identify  and  explicate  the  criteria  used  in  assessing  the 
"quality"  and  "relevance"  of  the  sample  set  of  projects. 

'^At  a  later  stage  in  the  work. 
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Using  a  modification  of  the  traditional  "peer  review"  procedure, 
which  calls  for  the  informed  judgment  of  competent  individuals  tp  evaluate 
the  "scientific  merit"  of  particular  research  Project  Proposals,  the  SOAP 
team  sought  also  to  emphasize  in  this  Exercise  the  concept  of  Program 
Evaluation.    They  did  this  by  including  separate  assessments  for  "tech- 
nical quality"  and  "program  relevance"  of  the  individual  projects  in 
order  to  help  the  U.S.  Bureau  of  Mines  determine  fufiding  priorities  for 
its  Energy  Research. 

•   For  this  Exercise*,  six  review  groups,  or  "juries,"  consisting 
each  of  three  to  five  members,  were  asked  to  rate  a  representative  sample 
of  research  Project  Proposals  submitted  to  the  Assistant  DirectQX -Bnejc gy , 
U.S.  Bureau  of  Mines,  for  FY-74  funding. 

With  Dr.  Ham's  help,  the  team  selected,  at  random,  seventeen  typical 
on-going  (FY  '73-74)  Project  Proposals  to  be  evaluated. 

Juries  selected  were: 


Jury  •  ;-  Research  Review  Groups 

I.  ..V;..  USBM  Washington  Staff 

II.  USBM  Washington  Staff 

III.        .■.  .  WASH/MERC/ BERC  Staff 

IV.  MERC  Research  Supervisors 

V.  .,  ..  ^  BERC  Research  Supervisors 

VI .  Industry  Technical  Advisory  Panel 


On  May  16,  1973,  three  of  the  SOAP  team  members**  met  with  the 
Project  Officer,  Harry  R.  Johnson,***  in  his  office  to  present  their  plan 
for  conducting  Exercise  I  and  to  gain  his  approval  for  administering  it. 
He  was  of  the  opinion  that  the  idea  was  "great,"  so  the  team  made  arrange- 
ments for  the  USBM  Washington  Staff  to  meet  the  following  day  at  Ballston 
Towers,  in  Arlington,  Virginia,  in  order  for  them  to  participate. 

-See  Project  SOAP  -  Exercise  I  Memorandum,  May  15,  1973,  Appendix  B-1  Thru  B-4. 
'  both^of^TAc!^  Arlington,  and  Robert  S.  Cutler  and  Dr.  Mary  Ellen  Golby, 

•*Leo  Schrider  was  also  present  that  day. 
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2.4    Exercise  I:    Preliminary  Assessment  of  Research  Project  Proposals 

Following  its  May  1st-  briefing  to  the  USBM  Research  Directors,  who 
were  meeting  in  Washington  for  a  Mid-Year  Review,  the  SOAP  team  was  urged 
to  conduct  a  test  of  a  management  tool  for  evaluating  projects  in  terms 
of  stated  program  goals.    What  was  desired  was  a  means  for  ascertaining: 
(a)  how  consistently  various  people  associated  with  the  USBM  Program  of 
Energy  Research  would  rate  on-going  Project  Proposals,  and  (b)  what  their 
rationale  was. 

A  preliminary  Exercise  was  then  designed  to  determine  the  feasibility 
of  obtaining  a  meaningful  assessment  of  the  "quality"  and  "relevance"  of 
individual  research  projects  (and  their  criteria),  by  requesting  selected 
review  groups  to  evaluate  a  representative  sample  of  research  Project 
Proposals  submitted  to  the  U.S.  Bureau  of  Mines  for  FY- 74  funding. 

Two  separate  assessment  scores  were  sought:    one  for  "technical 
quality,"  and  the  other  for  "program  relevance."    These  scores  would  later* 
be  used  as  a  basis  for  comparing  the  resulting  rank  order  for  the 
sample  set  of  projects  with  the  actual  experience  of  the  organization  in 
establishing  priorities  for  the  funding  of  research  projects. 

Specifically,  the  objectives  of  this  Preliminary  Exercise  were: 

•  To  determine  the  feasibility  and  acceptability  of  evaluating 
Energy  Research  Project  Proposals    separately,  for  "technical 
quality"  and  "program  relevance." 

•  To  analyze  the  scoring  patterns  of  specific  review  groups 

in  determining  the  "rele-vance"  of  particular  Energy  Research 
projects  to  the  new  Energy  Research  mission  of  the  U.S. 
Bureau  of  Mines. 

•  To  identify  and  explicate  the  criteria  used  in  assessing  the 
"quality"  and  "relevance"  of  the  sample  set  of  projects. 

*At  a  later  stage  in  the  work. 


27 


Using  a  modification  of  the  traditional  "peer  review"  procedure, 
which  calls  for  the  informed  judgment  of  con^etent  individuals  to  evaluate 
the  "scientific  merit"  of  particular  research  Project  Proposals,  the  SOAP 
team  sought  also  to  emphasize  in  this  Exercise  the  concept  of  Prpgram 
Evaluation.    They  did  this  by  including  separate  assessments  for  "tech- 
nical quality"  and  "program  relevance"  of  the  individual  projects  in 
order  to  help  the  U.S.  Bureau  of  Mines  determine  funding  priorities  for 
its  Energy  Research.       '  ■ 

■  For  this  Exercise*,  six  review  groups,  or  "juries,"  consisting 
each  of  three  to  five  members,  were  asked  to  rate  a  representative  sample 
of  research  Project  Proposals  submitted  to  the  Assistant  Director -Energy, 
U.S.  Bureau  of  Mines,  for  FY -74  funding. 

With  Dr.  Ham's  help,  the  team  selected,  at  random,  seventeen  typical 
on-going  (FY  '73-74)  Project  Proposals  to  be  evaluated. 
Juries  selected  were: 

Jury  Research  Review  Groups 

I.  USBM  Washington  Staff 

II.  USBM  Washington  Staff 

III.  r  WASH/MERC/BERC  Staff 

IV.  MERC  Research  Supervisors 

V.  BERC  Research  Supervisors 

VI .  Industry  Technical  Advisory  Pai>el 

On  May  16,  1973,  three  of  the  SOAP  team  members**  met  with  the 
Project  Officer,  Harry  R.  Johnson,***  in  his  office  to  present  their  plan 
for  conducting  Exercise  I  and  to  gain  his  approval  for  administering  it. 
He  was  of  the  opinion  that  the  idea  was  "great,"  so  the  team  made  arrange- 
ments for  the  USBM  Washington  Staff  to  meet  the  following  day  at  Ballston 
Towers,  in  Arlington,  Virginia,  in  order  for  them  to  participate. 

-See  Project  SOAP  -  Exercise  I  Memorandum,  May  15,  1973,  Appendix  B-1.  Thru  B-4. 
both^of^TAD!™  Arlington,  and  Robert  S.  Cutler  and  Dr.  Mary  Ellen  Colby, 

***Leo  Schrider  was  also  present  that  day. 
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2.4    Exercise  I:    Preliminary  Assessment  of  Research  Project  Proposals 

Following  its  May  1st.  briefing  to  the  USBM  Research  Directors,  who 
were  meeting  in  Washington  for  a  Mid-Year  Review,  the  SOAP  team  was  urged 
to  conduct  a  test  of  a  management  tool  for  evaluating  projects  in  terms 
of  stated  program  goals.    IVhat  was  desired  was  a  means  for  ascertaining: 
(a)  how  consistently  various  people  associated  with  the  USBM  Program  of 
Energy  Research  would  rate  on-going  Project  Proposals,  and  (b)  what  their 
rationale  was . 

A  preliminary  Exercise  was  then  designed  to  determine  the  feasibility 
of  obtaining  a  meaningful  assessment  of  the  "quality"  and  "relevance"  of 
individual  research  projects  (and  their  criteria),  by  requesting  selected 
review  groups  to  evaluate  a  representative  sample  of  research  Project 
Proposals  submitted  to  the  U.S.  Bureau  of  Mines  for  FY- 74  funding. 

Two  separate  assessment  scores  were  sought:    one  for  "technical 
quality,"  and  the  other  for  "program  relevance."    These  scores  would  later' 
be  used  as  a  basis  for  comparing  the  resulting  rank  order  for  the 
sample  set  of  projects  with  the  actual  experience  of  the  organization  in 
establishing  priorities  for  the  funding  of  research  projects. 

Specifically,  the  objectives  of  this  Preliminary  Exercise  were: 

•  To  determine  the  feasibility  and  acceptability  of  evaluating 
Energy  Research  Project  Proposals    separately,  for  "technical 
quality"  and  "program  relevance." 

•  To  analyze  the  scoring  patterns  of  specific  review  groups 

in  determining  the  "relevance"  of  particular  Energy  Research 
projects  to  the  new  Energy  Research  mission  of  the  U.S. 
Bureau  of  Mines. 

•  To  identify  and  explicate  the  criteria  used  in  assessing  the 
"quality"  and  "relevance"  of  the  sample  set  of  projects. 

*At  a  later  stage  in  the  work. 
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Using  a  modification  of  the  traditional  "peer  review"  procedure, 
which  calls  for  the  informed  judgment  of  competent  individuals  to  evaluate 
the  "scientific  merit"  of  particular  research  Project  Proposals,  the  SOAP 
team  sought  also  to  emphasize  in  this  Exercise  the  concept  of  Program 
Evaluation.    They  did  this  by  including  separate  assessments  for  "tech- 
nical quality"  and  "program  relevance"  of  the  individual  projects  in 
order  to  help  the  U.S.  Bureau  of  Mines  determine  funding  priorities  for 
its  Energy  Research.     .     ,  , '  ^  ; 

For  this  Exercise*,  six  review  groups,  or  "juries,"  consisting 
each  of  three  to  five  members,  were  asked  to  rate  a  representative  sajnple 
of  research  Project  Proposals  submitted  to  the  Assistant  Directpr -Energy, 
U.S.  Bureau  of  Mines,  for  FY-74  funding. 

With  Dr.  Ham's  help,  the  team  selected,  at  random,  seventeen  typical 
on-going  (FY  '73-74)  Project  Proposals  to  be  evaluated. 

Juries  selected  were:  .      :  ■ 

Jury  Research  Review  Groups 

I.  ..      .              .      USBM  Washington  Staff 

II.  USBM  Washington  Staff 

III.  :    ..    ■■    ■ -^r:          WASH/MERC/ BERC  Staff 

IV.  MERC  Research  Supervisors 

V.  BERC  Research  Supervisors 

VI .  Industry  Technical  Advisory  Panel 

On  May  16,  1973,  three  of  the  SOAP  team  members**  met  with  the 
Project  Officer,  Harry  R.  Johnson,***  in  his  office  to  present  their  plan 
for  conducting  Exercise  I  and  to  gain  his  approval  for  administering  it. 
He  was  of  the  opinion  that  the  idea  was  "great,"  so  the  team  made  arrange- 
ments for  the  USBM  Washington  Staff  to  meet  the  following  day  at  Bal^-Ston 
Towers,  in  Arlington,  Virginia,  in  order  for  them  to  participate. 

JSee  Project  SOAP  -  Exercise  I  Memorandum,  May  15,  1973,  Appendix  B-1    Thru  B-4. 

both^of'^TA?^  °^  Arlington,  and  Robert  S.  Cutler  and  Dr.  Mary  Ellen  Colby, 

**Leo  Schrider  was  also  present  that  day. 
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At  this  afternoon  meeting  on  May  17,*  the  details  of  Exercise  I 
were  explained    and  each  person  present  was  given  a  packet  of  materials 
which  included:    the  seventeen  Project  Proposals;  a  four -page  explanatory 
memorandum;  pertinent  material  excerpted  from  the  USBM  Mission  Statement 
to  serve  as  a  guide  for  judging  the  "relevance"  or  the  "quality"  of  these 
same  Project  Proposals;  and  a  tally  sheet.**  These  participants  were  asked 
to  mail  their  completed  tally  sheets  to  TAD/NBS  before  June  1,  1973. 

A  similar  orientation  briefing  by  the  Project  SOAP  team  was  given  to 
four  groups  at  BERC  (on  May  23)  and  four  groups  at  MERC  (on  May  31) .  In 
all,  67  individuals  were  asked  to  participate  in  this  research  evaluation 
exercise.    They  were  informed  of  the  background  and  purpose  for  requesting 
tills  type  of  subjective  assessnient,  and  each  member  of  each  "jury"  was 
asked  to  rate  the  sample  set  and  indicate  a  "priority  score"  based  upon 
his  own  judgment  of  the  facts  presented  in  the  proposal.    A  set  of  guide- 
line criteria  was  included  in  the  reference  materials.    It  was  stressed, 
each  time,  that  participation  in  the  Exercise  was  voluntary  and  that  if 
a  person  participated  but  wanted  to  remain  anonymous,  he  might. 

The  Research  Supervisors  at  both  Centers  (and  one  group  of  Staff) 
were  asked  to  rate  the  seventeen  sample  project  proposals  for  "program 
relevance"  only,  while  the  other  two  groups  of  Technical  Staff  were  asked  to 
rate  them  for  "technical  quality"  only.    A  table  summarizing  the  partic- 
ipation by  the  various  "juries"  is  included  in  Section  3.3***  of  this  report. 

^'Attending  were  Dick  Gooding,  Larry  Ramsey,  Larry  Burman,  Jerry  Ham, 
all  of  USBM,  Arlington;  Henry  Jacobson,  John  Tosh,  of  USBM,  Washington; 
and  Bob  Cutler  and  Mary  Ellen  Golby,  of  TAD.    Leo  Schrider  (who  is  not 
a  Monitor),  of  MERC,  also  attended.    Because  they  were  not  able  to  be 
present  at  this  meeting,  Alex  Mills,  Harry  Johnson,  Dick  Corey,  and 
Brian  Harney,  of  USBM,  Washington,  were  given  their  packets  the  preceding 
day. 

**See  Appendix  B. 
***See  Table  A,  p.  38. 
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3.    FINDINGS  AND  CONCLUSIONS 


The  primary  purpose  of  this  Exploratory  Survey  was  to  seek  clarification 
of  principal  issues  and  identification  of  some  of  the  actual  problems  asso- 
ciated with  implementing  miss ion -oriented  projects  within  the  Bureau  of  Mines 
Energy  Research  organization.    Such  exploration  and  clarification  of  desired 
"ends"  must  of  necessity  precede  the  analyses  of  possible  "means"  for 
improvement.  / 

The  Project  SOAP  team  conducted  a  series  of  14  group  interviews  at 
three  different  locations  within  the  USBM  Energy  Research  organization. 
These  included  meetings  with  key  personnel  at  both  the  Washington  Head- 
quarters and  at  MERC  and  BERC.    In  addition^  67  individuals  from  these 
groups  were  involved  in  a  preliminary  assessment  of  a  sample  set  of  Energy 
Researcli  Project  Proposals.    Tlie  purpose  was  to  test  a  program  management 
mechanism  for  evaluating  the  "quality"  and  "relevance"  of  individual 
projects  in  terms  of  stated  program  goals.     (No  analysis  of  the  results 
of  this  "Exercise  I"  is  included  in  this  report.) 

Tlie  follov/ing  represents  the  essence  of  tlie  findings  obtained  thus  far. 
Each  item  listed  is  the  result  of  many  man-hours  of  discussion  and  deliberation 
by  USBM  personnel  at  the  Centers  involved.       These  differ  from  the  listings 
shown  on  pages  19  and  24,  respectively,  in  that  they  represent  revisions  and 
acknowledgements  of  certain  "problem  areas"  identified  by  the  SOAP  team. 
Although  these  findings  are  not  all-inclusive,  they  clearly  indicate  a 
consensus  by  knowledgeable  people  on  some  of  the  program  management  problems 
that  are  believed  to  be  solvable  with  available  methods  and  personnel. 
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3.1    MERC  "Problem  Areas" 


Acknowledged  "Problem  Areas"  Associated  with  New  Management  Concepts 
Resulting  from  Second  Visit,  Morgantown  Energy  Research  Center, 

April  25-26,  1973 


1 .  Explicit  Mission  Statement 

•  Translation  of  Energy  Research  Program  "Goals" 
into  Research  Project  "Objectives" 

•  Participation  in  Program  Planning  Process 

•  Results -Oriented  Approach 

2 .  Improved  Organizational  "Feedback" 

•  Research  Supervisor  Function  Clarified 
(Intra -Communications:    within  the  Center) 

•  Research  Monitor  Function  Clarified 
(Inter-communications:    between  the  Centers) 

f    Strategic -Level  Planning/Evaluation  Staff 

3.  Mechanism  for  Evaluating  Projects  in  Terms  of  Stated 
Program  Goals 

•  Project  Priority  Determination 

.    "Quality"  (Peer  Review  Judgments) 

.    "Relevance"  (Internal/External  Opinions) 

Time -Frame  for  Results 
.    Costs  (multi-Year,  Total  Investment) 

•  Idea  Referral/Project  Proposal  System 
.  Stimulate  "Good"  Research  Projects 
.    Assure  Useful  Research  Results 

•  Projects  Proposal /Budget  Review  Process 
.    Targeted  Research  Activities 

.    Consistent  Evaluation  Criteria 

While  the  majority  of  the  men  at  MERC  originally  felt  that  an  explicit 
"Mission  Statement"  was  the  primary  thing  they  needed  to  give  direction 
to  their  work  (and  nearly  all  agreed  that  an  individual,  or  Station, 
Mission  Statement  could  be  subsumed  under  such  guidelines) ,  they  had  now  come 
to  recognize  that  the  Research  Supervisor's  function  had  greater  need  for 
clarification  than  that  of  the  Research  Mbnitor,  though  they  ranked  both 
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"problem  areas"  as  second  in  importance.    And  moreover,  after  considera- 
tion of  relative  priority,  they  felt  that  a  mechanism  for  judging  the 
"quality"  and  "relevance"  of  a  project  was  more  important  than  improving 
either  the  New  Idea  Referral  System  or  the  Project  Proposal  System. 

The  SOAP  team  reported  these  findings  at  the  Nfey  1st.    briefing  to 
the  USBM  Research  Center  Directors  and  others  from  the  USBM  Energy  group. 
3.2    BERC  "Problem  Areas" 

After  the  team's  visit  to  Bartlesville,  the  Center  Director, 
John  S.  Ball,  met  with  his  Research  Supervisors  to  draw  up  a  response 
to  the  list  of  "problem  areas"  identified  by  SOAP  and  presented  to  them 
for  reconsideration.    The  resulting  list  emerged: 

"Problem  Areas"  Acknowledged  as  a  Result  of 

Project  SOAP  Inifial  Visit  to  Bartlesville  Energy  Research  Center 
 Moy  21-23,  1973  

1.  Improved  Project  Manogement  Review  Mechanism 

•  Determinotion  of  "Technical  Quality"  and  "Progrom  Relevonce"of  Research 
Activities 

•  Continuing  Assessment  with  Feedb  %;k  of  Center  Priorities 

•  Assessment  of  Program  Results  as  a  Basis  for  Program  Evaluation 

2.  New  Idea  Development/Project  Proposal  System 

•  Present  Procedure  Vague  and  Ineffective 

•  Lock  of  Incentive  (Funding)  to  Stimulate  Innovation 

•  Requires  Meaningful  Feedback 

3.  Clorification  of  Headquarters  Energy  Division  Staff  Function 

•  Improved  Program  Planning/Evaluation  by  Staff 

•  Research  Monitor  Function  not  Clearly  Defined 

'         •      Lioison  with  Technical/Industry  Advisory  Groups 

4.  Recognition  of  Failure  of  Communication  Between  Washington  Headquarters  and 
Field  Stations  and  Between  Field  Stations 

•  Clarification  of  Research  Direction — Correlation  of  Policy  Positions  and 
Mission  Statement 

•  Acceptance  of  Specific  Energy  Research  Program  (and  Priorities)  from 
Heodquorters  by  Centers 

•  Proliferation  of  Reports  and  Lack  of  Evidence  of  Their  Usefulness 

•  Budgetary  Decision  Progress 

•  Inadequate  Funding  Support  to  BERC  Copcbilltles  in  Face  of  Current  Notional 
Energy  Crisis 
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While  at  Bartlesville,  the  SO^P  team  had  learned  that  BERC  differs 
as  much  from  MERC  as  the  Oklahoma  plains  differ  from  the  West  Virginia 
hills  or  as  oil  differs  from  coal.    One  sensed  that  the  Staff  at  BERC 
are  un.usually  strong  administrators  and  are  highly  individualistic.  One 
sensed,  too,  that  there  existed  at  BERC  the  suspicion,  frequently  iterated, 
that  the  Washington  Staff,  who  ars  primarily  ''coal  men,"  are  not  sensitive 
to  the  needs  of  the  "oil  men"  at  Bartlesville.    The  consensus  was  that 
the  top  decision-makers  at  Headquarters  are  "99%  coal  and  1%  petroleum." 
From  the  basis  of  a  balanced  Energy  Program,  there  appears  to  be  some  inequality  in 
the  distribution  of  funds  to  research  in  coal  and  petroleum:  petroleum 
provides  751  of  the  total  energy  used, while  coal  provides  only  25%,  yet 
coal  research  has  an  annual  appropriation  of  $12  million,  while  petroleum 
has  an  appropriation  of  only  $3  million,,    At  any  rate,  the  SOAP  team  learned 
that  BERC  is  definitely  oriented  to  one  commodity,  namely,  petroleum, 
although  they  did  note  that  several  coal  research  and  Health  and  Safety 
projects  are  being  conducted. 
3.3    Exercise  I  Preliminary  Results 

Essentially,  vAiat  the  SOAP  team  hoped  to  find  out  from  this  prelin- 
inary  exercise  was  whether  the  use  of  a  "peer- review"  mechanism  to  aid  in 
evaluating  Energy  research  projects  vms  operationally  feasible.    They  tested, 
by  obtaining  actual  ratings,  the  opinions  of  various  people  in  "the  system" 
who  were  asked  to  become  involved  in  a  simulation  of  this  decision  process. 

Of  the  67  professionals  involved,*  56  actually  participated  by  submitting 
their  tally  sheets.    Each  participant  rated  a  sample  of  17  Project  Proposals 
on  the  basis  of  his  own  concept  of  either  "tecbjiical  quality"  or  "program 
relevance,"  but  not  both.    Their  ratings  were  subjected  to  a  preliminary 
analysis  to  determine  if  there  exists  a  meaningful  and  consistent  concept 
underlying  each  of  these  two  criteria  dimensions  to  warrant  their  operational 
'^See  Table  A,  p.  38. 
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use  in  a  Program  Management  evaluation  tool.    (See  Figure  a.,  following 
page . ) 

The  findings  from  a  preliminary  analysis  of  this  first  Exercise 
are  quite  indicative  though,  of  course,  not  definitive: 

1.  There  was  implicit  in  the  participation  of  83.5%  of  the 
individuals  asked  to  serve  as  reviewers  (and  in  their  sub- 
sequent expressions  of  curiosity  about  and  interest  in  the 
results)  a  general  agreement  that  such  quantitative  methods 
for  evaluating  research  project  priorities  within  the  two 
USBM  Energy  Research  Centers  (MERC  and  BERC)    are  operationally 
feasible. 

2.  As  can  be  seen  from  Figure  a*,  which  presents  the  "modes" 
(most  frequent  rating  given  each  project  along  each  axis), 
there  is  some  obvious  agreement  anwng  those  reviewers  in 
applying  their  concepts  of  "program  relevance,"  but  not 

a  clear  consensus.     (The  concept  of  "program  relevance"  is  a 
complex  one  requiring  greater  specificity  and  communication  to 
assure  a  common  understanding  within  the  organization.) 

3.  The  greater  degree  of  consistency  among  the  reviewers  in  judging 
"technical  quality"  was  partially  due  to  the  clearer  definition 
of  this  concept  and  also  the  fact  that  the  sample  set  were 
on-going  projects.    The  concept  underlying  "technical  quality" 
as  used  in  this  Exercise  was  found  to  include  more  than  simple 
technological  considerations.    These  additional  criteria  appear 
to  be  closely  related  to  priority  considerations. 

4.  If  the  reviewers  had  experienced  great  difficulty  in  separating 
"relevance"  from  "quality"  considerations,  the  plotted  points 
would  have  been  clustered  around  the  "45  degree"  diagonal  line 
in  Figure  a.    Tliis  is  clearly  not  the  case;  the  distinction 
between  the  two  criteria  is  one  which  can  be  rendered  operational 
in  such  a  rating  procedure  and,  indeed,    people  at  two  different 
Research  Centers  are  capable  of  concurring  that  a  particular 
project  (here,  "Project  G")  ranks  quite  high  on  one  scale  but 
low  on  the  other. 

5.  It  is  interesting  to  note  that  Project  M  (an  "oil"  Project 
from  BERC)  was  judged  by  the  MERC  people  to  be  more  "relevant" 
than  by  BERC,  and  similarly,  Project  B  (a  "coal"  Project  from 
MERC)  was  considered  more  relevant  by  the  reviewers  at  BERC 
than  at  MERC.    This  finding  suggests  that  the  reviewers  can 
indeed  broaden  their  concept  of  mission  to  include  the  goals 
of  the  Energy  Research  Program  as  a  whole,  when  evaluating  the 
particular  merits  of  projects  originating  from  USBM  Research 
Centers  other  than  their  own.     (See  Figures  b.  and  c,  pages  36,  37.) 


*And  its  comparison  with  Figures  b.  and  c.  (See   pp.     35,36,  and  37.) 
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TABLE  A 


Exercise  I  Results 
Summary  of  Participant  Responses 
(As  of  June  10,  1973) 


JURY 

PARTICIPANT 

RESPONSES 

PERCEOTAGE 

RECEIVED 

I.    USBM  WASH  Staff 

5 

5 

!  1001 

II.    USm  WASH  Staff 

.  4 

3 

751 

"II.    WASH/^4ERC/BERC  Staff 

a.      WAon     J  y.)ucixxL.^j 

A 

3 

75% 

b.    MERC  #1  (Relevance) 

5 

2 

40% 

c.    McKL  ffz  ^l^ualltyJ 

0 

3 

50% 

d.    MERC  #3  (Quality) 

5 

4 

80% 

e.    DcKL  ffi  ij^uaiityj 

5 

4 

80% 

f.    BERC  #2  (Quality) 

5 

100% 

g.    BERC  #3  (Relevance) 

5 

5 

100% 

IV.    MERC  Supervisors 

7 

7 

100% 

V .     dckLj  oupervisors 

c 
0 

£: 
0 

T  n  n  Q- 
100-5 

VI.     Industry  Advisory  Panels 

a.     Industrial  Members 

5 

5 

100% 

b.    Academic  Members 

5 

4 

80% 

TOTAL 

67 

56 

83.5% 

Other  Summary  Data: 


1.    No.  Criteria  Statements  Received  46 


2.    No.  Signatures  Received  36 


Project  SOA  I*  ! 
TeehniMi  Analysis  Division 
Nofional  Bareau  of  Standards 

RSC/RMC  j 
10  June  1974 
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The  remarks  received  from  participants  are  noteworthy.    In  describ- 
ing his  reaction  to  Exercise  I,  one  BERC  Project  Leader  ivrote: 

The  most  obvious  result    of  participation  in  this  project  was  the 
realization  of  how  little  actually  useful  information  is  contained 
in  tlie  typical  project  proposal.    Some  16  proposals,  totaling  over  5 
million  dollars,  averaged  less  than  two  pages  each,  of  which  less 
than  a  half  page  is  devoted  to  a  very  brief  outline  of  the  work  to 
be  accomplished  with  the  capabilities  of  the  principal  investigator 
and  his  available  facilities,  arriving  at  an  intelligent  appraisal  of 
the  potential  for  success  of  a  particular  project  is  essentially 
impossible  with  the  amount  of  information  supplied.    Even  allowing  for 
the  briefness  of  the  presentation,  the  proposals  are  inadequate  for 
the  most  part.    As  an  example,  tlie  objective  of  a  research  project  is 

not  (or  should  not  be)  "to  study  the  composition   ",  it  should 

be  to  make  specific  determinations  of  some  facts  as  a  result  of  study 
and  physical  experiment.    The  objectives  should  state  what  is  intended 
for  accomplishment  during  the  specific  period,  not  simply  broad  in- 
definite, hoped  for  results  such  as  "increased  recovery  of  oil". 


Another  participant  had  difficulty  separating  his  concept  of  "program 
relevance"  from  that  of  "technical  quality."    He  wrote: 

I  found  it  quite  difficult  to  determine  "Program  relevance" 
for  the  selected  projects  without  also  considering  technical 
feasibility,  as  technical  feasibility  is  dependent  on  other 
parameters  such  as  funding,  timing,  competence,  manpower.... 

Further,  he  went  on  to  add: 

The  resulting  rating  should  be  recognized  as  only  a  partial 
evaluation  of  the  priority  of  a  project.    Likewise,  a  technical 
feasibility  evaluation  also  is  only  a  partial  evaluation.... 
What  I'm  leading  up  to  is  the  difficult  and  less  "clear  cut" 
subjective  judgment  that  must  be  used  in  selecting  the 
"mix"  of  short-,  intermediate-,  and  long-term  projects  as  well 
as  the  mix  of  basic,  development,  and  applied  research  that 
must  be  maintained  for  a  balanced  program.    I  realize 
determining  the  ratios  of  the  mix  may  be  arbitrary  decisions, 
but  they  must  be  made  to  assure  an  adequate  background  of 
fundamental  and  developmental  work  and  competence  necessary  for 
the  pursuit  of  long-term  projects.    To  change  the  present  situ- 
ation would  require  a  skillful  surgeon,  a  sharp  knife,  and 
probably  a  considerable  degree  of  pain.    The  USBM  Energy  Group 
appears  to  suffer  from  a  number  of  illnesses  that  compound  the 
problem.    For  example,  there  are  too  many  Chiefs  and  not 
enough  Indians.    Many  of  the  Directors,  both  in  the  Field  and 
at  the  Washington  Office,  are  required  to  operate  with  staffs 
that  are  too  small  and  as  a  result  are  believed  by  some  to  be 
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incapable  of  thorough  analysis  and  coordination  o£  research 
projects  before  they  are  implemented.    There  seems  to  be 
pressure  to  "get  into  the  field"  with  projects  before  ade- 
quate laboratory  research  work  is  completed,  apparently 
because  of  the  better  visibility  of  field  research. 

The  basic  missing  ingredient  is  specific  program  direction  from 
the  Washington  office.    Because  of  the  lack  of  it,  most  pro- 
jects are  written  on  the  ba'sis  of  what  the  researcher  would 
like  to  do  rather  than  what,  needs  to  be  done.    The  simple 
suggestion  to  "improve  the  efficiency  of  oil  recovery"  is 
not  enough.    Field  personnel  are  too  restricted  in  their 
viewing  area,  in  their  work  experience,  and  in  their  contacts 
with  Industry  (except  in  a  few  special  cases) ,  to  effectively 
relate  their  capabilities  with  broad  needs.    They  either  over 
estimate  the  amount  of  research  that  can  be  accomplished  at  a 
given  funding  level  and  consequently  spread  themselves  too 
thinly,  or  they  concentrate  all  effort  on  a  minuscule  portion 
of  the  real  problem,  either  through  lack  of  understanding  of  the 
problem  or  because  of  personal  reasons  of  self  interest.  Only 
direct,  detailed,  and  coordinated  program  direction  from  an 
informed  and  concerned  Washington  Staff  can  correct  this 
situation. . . . 


One  person  at  MERC  summed  it  up  when  he  said:    "the  Washington 
Office  has  exercised  little  in  the  way  of  guidelines  for  Energy  programs 
in  the  past  and  has  had  little  or  no  time  to  formulate  thoughts  on 
any  program  or  to  evaluate  what  is  submitted  to  them  from  the  field. 
Yet,  the  researchers  annually  look  to  them  for  some  relatively  defined 
goals  towards  which  efforts  are  to  be  directed.    In  this  time  of 
concern  over  energy  shortages,  however,  perhaps  we  need  a  NASA- type 
'systems  approach'  to  accomplish  some  important  development." 
3.4  Conclusions 

Thus  far  into  this  exploratory  survey,  the  Project  SOAP  task  group 
has  followed  a  planned  fact-finding  procedure.    Based  upon  interviews  at 
three  locations  (WASH,  MERC,  BERC) ,  it  has  focused  attention  on  principal 
problems  observed  (and  verified)  at  each  of  the  operating  units  visited. 
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Of  the  many  problems  the  USBM  management  has  been  aware  of,  the 
lists  of  acknowledged  "problem  areas,"  identified  in  Sections  3.1  and  3.2 
above,  represent  those  where  reasonable  improvements  are  considered  possi- 
ble, both  by    the  SOAP  team  and  by  the  Field  Center  personnel  themselves. 
These  lists  include  a  recognition  from  two  Research  Centers  of  the  need 
for  some  rational  mechanism  for  evaluating  the  quality  and  relevance  of 
individual  projects  for  use  in  determining  project  priorities  at  the  Field 
Station  level. 

At  this  early  stage,  the  SOAP  team  (including  members  of  the  USBM 
Washington  Staff)  have  concluded  that  two  of  the  key  functions  of  the 
decision-making  "pipeline"  in  most  need  o£  attention  are  those  of  the 
Research  Supervisor  and  the  Research  Monitor.    There  is  apparently  confusion 
at  both  the  Headquarters  and  Field  levels  concerning  what  is  or  ought  to  be 
their  respective  roles  in  bringing  about  tjhe  new  concept  of  miss  ion -oriented 
program  management. 

As  for  the  existing  "Idea  Referral  System,"  the  procedure  appears  to 
some  at  both  MERC  and  BERC  to  have  become  an  effective  process  for 
killing  "good  ideas."    There  is  a  need  for  more  dialogue  between  the 
Washington  Headquarters,  who  should  establish  program  "requirements,"  and 
the  Field  Centers,  whose  functions  ought  to  emphasize  matters  of  technical 
feasibility.    However,  if  each  Research  Center  Director  is  to  be  responsible 
for  establishing  budget  priorities,  questions  concerning  the  comparative 
"program  relevance"  of  particular  "ideas"  which  would  be  in  competition  with 
on-going  projects  will  have  to  be  resolved.    As  it  is,  there  is  little 
incentive,  and  much  difficulty,  in  motivating  individuals  to  propose  new 
projects  from  the  Field  level. 
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Perhaps  this  is  the  main  beneficial  result  of  the  Project  SOAP 
Exploratory  Survey:    the  gaining  of  insights  by  the  people  within  "the 
system"  about  precisely  what  it  is  they  believe  is  wrong,  and  what  ought 
to  be  done  by  way  of  remedy.    One  of  the  facts  generally  recognized  about 
operational  improvement  is  that  the  remedy  cannot  work  unless  most  of  the 
people  involved  are  convinced  th^t  it  will  work.    There  is,  we  have  found, 
much  more  to  implementing  a  new  concept  of  Program  Management  than  systems 
analysis  alone  can  provide. 

The  SOAP  team  concluded  from  its  preliminary  interpretation  of  the  results 
of  Exercise  I  that  the  mechanism  sijijulated  for  determining  priorities  based 
on  separate  assessments  of  "quality*'  and  "relevance"  appears  to  provide 
a  reasonable  basis  for  further  work^  at  least  for  MERC  and  BERC.    That  is, 
the  development  of  a  management  tool  for  determining  the  priorities  of 
individual  projects  at  the  Center  level  upon  which  to  base  program  priorities 
for  Mid -Year  Budget  Review  and  for  evaluating  New  Idea  Proposals  seems 
feasible.    How^ever,  more  work  is  needed  before  this  evaluation  mechanism 
can  be  useful  as  an  indicator  of  specific  program  performance*  based  upon 
the  two  generalized  research  management  criteria:    teclmical  excellence  and 
relevant  results.  , 

Certain  changes  in  this  mechanism  are  suggested.    They  include  the 
forma.tion  of  more  sharply  contrasting  "peer  group"  juries;  the  translation 
of  Mission  Statement  "goals"  into  achievable  Energy  Program  "objectives"; 
and  improvement  of  the  Project  Proposal  descriptions  themselves.  These 
procedural  items  can  be  worked  out  in  practice. 

'^The  former  is  judged  typically  by  professional  peers,  and  the  latte.r  by 
Budget  Administrators,  Congressmen  and  various  people  within  particular 
industrial  research  organizations . 
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A  second  simulation  exercise  is  recommended  incorporating  the  above 
improvements  but  also  involving  all  six  USBM  Energy  Research  Centers  and 
dealing  with  actual  projects  (both  on-going  and  new  proposals)  to  be' 
evaluated  at  tlie  forthcoming  Mid-Year  Budget  Review  in  October  1973. 
It  is  anticipated  that  decision -makers  at  various  levels  in  the  Bureau  of 
Mines'  organization  will  then  be  able  to  obtain  a  more  precise  picture 
of  what  is  happening  within  their  overall  program  of  Energy  Research. 
Further  development  and  use  of  this  '"peer  review"  mechanism  would  provide 
a  new  basis  for  making  decisions  about  program  priorities  and  strategies 
effecting  a  more  rational  balance  in  research  resource  allocations. 
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March  9,  1975 


Mr.  V/illicun  L.  Crentz 
Assistant  Director  -  Energy 
U.S.  Ixireau  o£  Mines 
Wasliington,  D.  C.  20240 

Dear  Kr.  Crentz: 

As  a  result  of  the  March  6  meeting  at  the  National  Bureau  o£  Standards 
between  Harry  Johnson,,  Leo  Schrider,  and  /\ridy  Decora  of  your  staff  and 
members  of  our  Project  SO.-\P  Team,  v;e  are  pleased  to  assist  your  Energy 
Group  staff  in  implementing  the  new  mission  and  management  concepts 
associated  with  the  National  Energ)'  Program. 

We  understand  that  your  primary  interest  is  improving  communications 
between  Headquarters  and  field  levels.    This  would  be  useful  for  developing 
more  effective  management  controls  for  directing  individual  research  activ- 
ities toward  the  new  goals  defined  by  the  Energy  Research  Program.  In 
this  regard  you  would  like  us  to  make' an  objective  examination  of  functional 
operating  activities  and  infonriation  flows  at  various  decision  levels  within 
the  organization.    Such  an  examination  will  serve  to  clarify  certain  issues 
concerning  your  broader  mission  and  new  evaluation  requirements  for  mid- 
yeai-  project  reporting. 

Of  secondary  concern,  but  possibly  the  simpler  problem  to  address,  is  the 
need  for  a  practical  mLechanism  for  evaluating  both  major  program  segments 
^d  individual  research  projects  in  terms  of  program  goals.    Because  of 
the  large  numiber  of  ongoing  projects  and'  the  many  new  and  potentially 
useful  proposals  wnich  come  to  the  Research  Center  Directors  as  contract 
proposals  (or  from,  the  research  interests  of  individual  scientists  and 
engineers  v/ithin  their  field  research  centers)  a  systematic  m.eans  for 
determining  "technical  quality"  and  "program  relevance"  is  an  important 
management  tool  for  them  to  use  when  approving  new  research  projects  and 
for  establishing  priorities.    Tliese  program  relevance  values  can  also  be 
used  as  criteria  for  integrating  special  emphasis  research  areas,  for 
making  inter-program  trade-offs,  and  for  providing  visible  evidence  of  the 
coherence  of  program  management  at  all  levels. 

In  order  to  be  of  real  help  to  your  Program's  staff,  we  must  first  clarify 
the  principal  issues  and  identify  the  actual  probJems  that  currently  exist. 
One  of  these  central  issues,  as  Harry  and  Leo  described  it  to  us,  is  one 
of  communication  of  new  ideas  between  Headquarters  and  the  field- -m.ore 
specifically,  between  the  individual  researchers,  who  are  the  main  strength 


of  the  energy  program,  and  top  nianagcn.ent  personnel  in  Washington.  Further 
we  were  informed,  there  is  a  need  for  improving  vertical  infonrntion  flow, 
and  horizontal  communication  (cross -talk)  between  individual  scientists  at 
one  field  center  wirh  those  of  other  centers  and  bot^.veen  individual  project 
leaders  aiid  research  supervisors  with  their  peers  elsewhere.  TliereforCj 
before  undertaking  prescription  of  any  particular  methods  for  improving 
the  accuracy  and  precision  of  feedback  infonriation,  we  believe,  as  a 
result  of  our  experience,  that  a  careful  examination  must  be  made  to 
assure  that  the  desired  benefits  to  the  organization  would  result. 

In  short,  we  propose  to  make  a  three-month  exploratory  suivey  of  your 
"Prograjn  Management  System."    V/e  hope  to  develop,  for  your  staffs  a  basis 
for  objectively  determining  the  nature  and  extent  of  the  "commrunication 
problem"  at  each  level,    Tliis  initial  phase  should  also  provide  you  with 
a  basis  for  considering  the  feasibility  of  extending  this  approach, 
•  leading  ultimately  toward  your  organization's  iniplementing  a  quantitative 
method  for  improved  management  control. 

We  are  ready  to  start  this  assignment  on  March  19,  1973,.  and  plan  to 
complete  the  work  by  June  30,  1973.    Participation  from  your  staff  is, 
of  course,  required  as  soon  as  possible,  so  that  cur  necessary  orienta- 
tion and  survey  planning  can  proceed  on  schedule. 

Upon  receipt  of  a  formal  work  order  in  the  amount  of  $37,400,  for  reimburse 
Ficnt  of  the  estimated  salary/  and  expenses  indicated  on  the  proposed  budget  j 
this  project  will  be  authorized.  If  ■:here  are  further  questions  or  related 
matters,  please  let  me  know. 

We  look  forward  to  working  closely  with  your  staff  on  this  interesting 
and  important  assignment. 

Sincerely, 


Louis  C.  Santone,  Acting  Chief 
Technical  Analysis  Division 
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Ur.  R.  S.  Cutler,  NBS,  6/1/73 

Under  New  Idea  Development,  the  idea  referral  system  appears  to  be  an  effective 
system  for  killing  ideas.   The  Issue  Paper  route  is  better  but  ill-defined,  and 
normally  to  be  used  only  on  invitation  from  the  Washington  Office.    There  is 
little  incentive  to  develop  a  new  program  if  the  reply  is  to  kill  some  present 
programs  in  order  to  reprogram  funds.   The  development  of  strong  programs  re- 
quires a  dialogue  between  the  Washington  Office  who  establish  program  relevance 
and  the  field  whose  function  is  to  determine  technical  feasibility. 

The  Clarification  of  the  Headquarters  Staff  Function  would  perhaps  help  to  focus 
discussion  where  it  would  do  the  most  good.   We  are  rather  uncertain  as  to  what 
mix  of  monitoring  functions  and  purely  staff  operations  are  desired.  Manuscripts 
and  convention  travel  requests  go  directly  to  staff  while  proposals  and  manage- 
ment review  reports  apparently  are  not  staff  functions.   Some  industry  advisory 
groups  have  fiaison  through  the  Washington  Office  while  others  are  directly  with 
the  Research  Centers. 

The  section  labeled  "Failure  of  Communication"  was  changed  from  "Communication 
Gap"  because  we  believe  it  is  not  a  problem  of  speaking  different  languages  but 
rather  of  failing  to  speak  at  all.   This  is  understandable  with  the  urgency  and  con- 
fusion, but  it  does  at  times  present  problems.   The  positions  presented  by  the  Project 
SOAP  team  as  representing  Washington  thinking  did  not  always  match  our  under- 
standing.  Specifically  we  found  some  differences  between  the  Mission  Statement 
and  our  understanding  of  the  energy  research  policy.  We  were  surprised  to  hear  that 
Washington  regarded  our  acceptance  of  programs  and  priorities  as  less  than  complete. 
We  do  feel  that  reporting  requirements  have  been  increased  and  wish  we  had  more 
feedback  to  indicate  that  they  are  being  used.   We  are  particularly  uninformed  as 
to  budgetary  decisions  and  are  unable  to  relate  such  decisions  to  the  preparation  of 
further  proposals.   We  feel  that  the  present  energy  crisis  is  the  first  opportunity  in 
many  years  to  increase  our  usefulness  and  are  dismayed  that  instead  we  face  a 
reduction. 

I  believe  that  the  visit  of  the  SOAP  team  was  beneficial  in  that  it  caused  us  to 
stop  and  assess  our  problems.    In  particular,  the  exercise  of  rating  projects  brought 


A-S 


3 


Ur.  R.  S.  Cutler,  NBS,  6/1/73 


home  to  some  of  our  people  the  comoetition  that  they  face.   We  wili  look  forword 
with  interest  to  your  report  of  June  30,,    If  there  is  additional  informotscri  v/e  can 
furnish,  call  or  write  us. 

Sincerely  yours. 


John  S.  Bali 
Research  Director 

Enclosure 
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Problem  Areas  Identified  as  a  Result  of 
Project  SOAP  Initial  Visifr  to  Bartlesville  Energy  Research  Center 

 May  21-23,  1973  


Improved  Project  Management  Review  Mechanism 

•  Determination  of  "Technica!  Quality"  and  "Program  Relevance"  of  Research 
Activities 

•  Continuing  Assessment  with  Feedback  of  Center  Priorities 

0      Assessment  of  Program  Results  as  a  Basis  for  Program  Evaluation 

New  Idea  Development/Project  Proposal  System 

•  Present  Procedure  Vague  and  Ineffective 

•  Lack  of  Incentive  (Funding)  to  Stimulate  Innovation 

•  Requires  Meaningful  Feedback 

Clarification  of  Headquarters  Energy  Division  Staff  Function 

•  Improved  Program  Planning/Evaluation  by  Staff 

•  Research  Monitor  Function  not  Clearly  Defined 

•  Liaison  with  Technical/Industry  Advisory  Groups 

i 

Recognition  of  Failure  of  Communication  Between  Washington  Headquarters  and 
Field  Stations  and  Between  Fseld  Stations 

•  Clarification  of  Research  Direction — Correlation  of  Policy  Positions  and 
Mission  Statement 

•  Acceptance  of  Specific  Energy  Research  Program  (and  Priorities)  from 
Headquarters  by  Centers 

•  Proliferation  of  Reports  and  Lack  of  Evidence  of  Their  Usefulness 

•  Budgetary  Decision  Progress 

•  Inadequate  Funding  Support  to  BERC  Ccpabilities  in  Face  of  Current  National 
Energy  Crisis 


A- 7 


ii-PPEl\T)IX 


(Section  Bj 


Exercise  I  Materials 
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For  this  preliminary  Exercise,  six  review  groups,  or  "juries," 
consisting  each  of  three  to  five  members  will  rate  a  representative 
san^le  of  research  project  proposals  submitted  to  the  Assistant 
Director-Energy,  U.  S.  Bureau  of  Mines,  for  FY-74  funding.  The 
juries  selected  are  as  follows: 


Juries  Research  Review  Groups 

I .  USBM  Washington  Staff 

II.  USEM  Washington  Staff 

III.  MERC  Research  Supervisors 

IV.  •                            WASH/MERC/BERC  Staff 

V.  BERC  Research  Supervisors 

VI.  '                                 Industry  Technical  Advisory 

Panel 


Procedure : 


Following  the  orientation  briefing  by  Project  SOAP  team  describing 
the  background  and  purpose  of  this  preliminary  Exercise,  each  member  of 
each  "Jury"  will  rate  the  sample  set  and  indicate  a  "priority  score" 
based  upon  his  own  judgment  of  the  facts  and  information  presented,  in 
accordance  with  the  appropriate  "criteria"  described  below. 

Juries  I,  II,  IV,  and  V  will  rate  each  project  for  "program  rele- 
vance" only,  \A\ile  Juries  III  and  VI  will  rate  "technical  quality"  only. 

Each  member  will  register  his  vote  (priority  score)  on  the  tally 
sheet  si:5)plied.    This  will  indicate  the  Juror's  professional  opinion  of 
the  particular  Research  Project  rated.    Scoring  will  consist  of  selecting 
a  number  between  1.0  to  5.0  (or  N/R)  where  1.0  signifies  HIGHEST  value 
and  5.0  Lowest  value,  with  N/R  meaning  "not  qualified  to  render  judgment." 
(0.5  increments  are  allowed.) 

After  each  reviewer  has  completed  his  rating,  he  is  to  send  the 
tally  sheet  only  to  NBS  in  the  si:pplied  pre-addressed  envelope. 

Criteria: 

For  this  Exercise,  "Technical  Quality"  refers  to  the  probability  of 
achieving  the  stated  research  objective,  talcing  into  account: 

1.  Qualifications  of  Principle  Investigator. 

2.  Availability  of  Coirpetent  Staff/Facilities. 
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3.  Scope  of  Work/Research  Cbjectives. 

4.  Technical  Approach/Work  Schedule. 

5.  Adequacy  of  Proposed  Budget. 

In  addition  to  the  above  necessary  technical  considerations , 
"program  relevance"  is  intended  to  be  used  as  a  variable  measure  for 
estimating  the  utility  (or  "worth-'whileness")  of  the  proposed  research 
results  for  which  no  generally  accepted  standards  or  criteria  has  been 
established.    In  other  words,  assuming  the  particular  research  project 
is  successful,  how  important  would  it  be  toward  meeting  the  goals  of 
the  Bureau  of  Mines  Energy  Research  Program  (as  described  in  the  Energy 
Research  Mission  Statement,  May  25,  1972). 

It  has  been  possible  for  many  people  (Research  Scientists,  Adminis- 
trators, Congressmen,  Industrial  Executives),  to  render  judgments  about 
"program  relevance,"  based  upon  various  criteria.    It  has  been  difficult, 
however,  for  the  Research  Scientists  to  explicate  these  'TDenefits"  of 
proposed  research  output,  it  is  not  clear  to  Budget  examiners  within 
the  Department  of  Interior  (and  The  Office  of  Budget  and  Management) 
whether  similar  criteria  are  being  used  by  the  various  people,  particu- 
larly those  with  obviously  different  backgrounds,  experience,  and 
interests.    Moreover,  the  importance  of  evaluating  Energy  Research 
"results"  in  terms  of  meeting  the  needs  of  the  Energy  Crisis  (and 
admittedly  these  "results"  are  difficult  to  define)  is  often  lost  in 
this  process . 

For  consideration  by  participants  in  this  Exercise,  the  following 
perspective  is  offered: 

"To  establish  a  conceptual  framework  for  our  role,  research 
might  be  viewed  as  a  continuous  flow  of  infomation  that  begins 
with  basic  discoveries  and  continues  through  commercial  develop- 
ment....The  problem- oriented  research  we  conduct  is  both  useful 
and  essential  for  the  development  of  new  processes  (of  Energy 
Production)  needed  by  our  society. ...  In  delineating  major 
Research  activities  to  be  undertaken  by  Energy  Research,  and 
relative  priorities,  we  have  been  guided  by:    (1)  the  national 
energy  policy,  (2)  technological  needs,  and  (3)  our  concept  of 
mission. ..." 

These  statements  should  be  regarded  as  a  framework  for  establishing 
a  common  base  for  evaluating  the  proposed  results  of  Energy  Research- 
conducted  by  the  U.  S.  Bureau  of  Mines  in  advancing  production  technology 
of  coal,  oil  shale,  petroleum  and  natural  gas. 
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Results  Expected: 

Based  on  the  analyses  of  the  priority  scores  received  from  the  six 
Jury  groups,  a  series  of  tabulations  and  charts  will  be  developed  to 
indicate: 

1.  Any  significant  differences  or  agreement  between  the 
various  review  groups  in  assessing  "program  relevance" 
of  the  particular  research  projects  rated. 

2.  Coirputed  rank  order  listing  of  sample  set  of  research 
projects,  based  on  mathematical  combination  of  technical 

:       "quality"  and  "program  relevance." 

3.  Degree  of  acceptability  (in  principle  or  practice,  as 
indicated  by  abstentions)  with  this  mechanism  for 
evaluating  energy  research  project  proposals  within  the 

.  'O'    ■  U.  S.  Bureau  of  Mines. 

4.  List  of  explicit  "program  relevance"  criteria. 


Project  SOAP 
Teehnieti  Analysis  Division 
National  Buroao  of  Standards 

RSC/RMC 
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Project  SOAP 
Exercise  I  Tally  Sheet 


ponc 
UJuh 

( 

'DDTADTTV  CrnDP" 
r  KIUKl  1 1  oLUKJc 

:Scale:  1.0-5.0) 

"  , 

(Basis  for  Judgment) 

A 

Stimulating  Tar  Sand 
Production 

B 

Formcoke 

C 

Earth  Fracture  Systems -- 
Application  to  Hydrocarbon 
Recovery 

D 

Utilization  of  Solid 
Wastes  from  Combustion 
and  MinincT  of  Coal 

E 

Studies  of  Asphalt  and 
Asphaltic  Materials 

F 

Reduced  S0->  in  Coni)Ustion 
Gases 

G 

Demonstration  of  Improved 
Secondary  and  Tertiary 
Methods  of  Recovering 
Petroleum 

H 

Coal  Disproportionation 
by  Catal\tic  Hydro genati on 

I 

Factors  Affecting  New  Oil 
Recovery 

1 

J 

Development  of  Marginal  Gas 
Resources  Utilizing  Natural 
and  Indiicpd  FractiiTPS 

K 

Corrosion  of  Fireside  Sur- 
faces in  Coal- Fired  Boilers 

L 

Fuels  and  Engine  Systems  to 
Enhance  Fuel  Economy  in  Clean 
Air  Cars 

M 

Waste  Oil  Recycling 

N 

Pressurized  Gas  Producer  and 
Gas  CleanuD  and  Purification 

0 

Development  of  a  "Prebum" 
Propellant  for  in  situ 
Underground  Coal  Gasification 

P 

v 

Demonstration  of  ^tethane  - 
Methanol  Coal  Conversion  Plani 

t 

Q 

Underground  Gasification  of 
Coal 

Jury  No.  Name  (optional) 
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ATTACHMENT  I 


U.  S.  BUREAU  OF  MINES 
ENERGY  RESARCH  PROGRAM 
MISSION  STATBIENT  - 

May  25,  1972 


(Annotated  Excerpts  from  Mission 
Statement  for  reference  by  Exercise  I 
Participants  -  5/15/73) 
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POLICY  IMPLEMENTATION 


The  specific  actions  by  the  Department  of  the  Interior  and  the 

Bureau  of  Mines  to  implement  President  Nixon's  Clean  Energy  Policy  are 

guided  by  the  President  and  by  the  directives  of  Congress.     The  policy 

of  Congress,   embodied  in  various  laws,  has  recently  been  supplemented 

by  enactment  of  the  Mining  and  Minerals  Policy  Act  of  1970  (84  Stat. 

1876).     That  act  states: 

.   .   .  it  is  the  continuing  policy  of  the  Federal  Government  in  the 
national  interest  to  foster  and  encourage  private  enterprise  in 

(1)  the  development  of  economically  sound  and  stable  domestic 
mining,  minerals,  metal,   and  mineral  reclamation  industries, 

(2)  the  orderly  and  economic  development  of  domestic  mineral 
resources,   reserves,   and  reclamation  of  metals  and  minerals  to 
help  assure  satisfaction  of  industrial,   security,   and  environ- 
mental needs,    (3)  mining,  mineral,  and  metallurgical  research, 
including  the  use  and  recycling  of  scrap  to  promote  the  wise  and 
efficient  uso  of  our  natural  and  reclaimable  mineral  resources, 
and  (4)  the  study  and  development  of  methods  for  the  disposal, 
control,   and  reclamation  of  mineral  waste  products,   and  the 
reclamation  of  mined  land,    so  as  to  lessen  any  adverse  impact  of 
mineral  extraction  and  processing  upon  the  physical  environment 
that  may  result  from  mining  or  mineral  activities. 

The  Secretary  of  the  Interior  has  been  assigned  the  responsibility  to 

carry  out  these  policy  directives  in  exercising  his  legal  authority. 

To  implement  this  responsibility,    the  Interior  Department  has 

established  the  following  energy  and  minerals  mission: 

To  assure  a  continuing  and  adequate  flow  of  energy  and  minerals 
from  reliable  sources  at  the  lowest  cost  consistent  with  the 
attainment  of  social  and  environmental  objectives 

and  has  charged  the  Bureau  of  Mines  with  the  responsibility: 

To  conduct  those  programs  of  inquiry  and  regulation  necessary  to 
keep  the  Government  informed  and  to  stimulate  the  private  sector 
to  produce  the  minerals  and  fuels  needed  to  supply  an  appropriate 
and  substantial  share  of  the  national  needs  in  a  manner  acceptable 
to  the  public  interest. 

To  help  discharge  these  responsibilities,  'the  mission  of  Energy  Research 

within  the  Bureau  of  Mines  has  been  formulated  as  follows: 

To  develop  effective  means  that  will  increase  the  availability 
and  improve  the  utilization  of  the  Nation's  energy  resources  at 
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reasonable  costs  and  in  a  manner  that  will  help  attain  desirable 
social  and  environmental  improvements. 

Specific  areas  of  research  are  conducted  at  six  energy  research 
facilities  which  employ  more  than  900  people.     The  location,  field 
manager,   and  mission  for  each  of  these  facilities  are  given  in  Appendix 
A.     Priorities  for  the  research  are  based  upon  anticipated  patterns  of 
energy  consumption  and  the  technologic  needs  as  subsequently  outlined 
in  this  statement. 
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TECHNOLOGIC  NEEDS-' 


The  production  and  consumption  of  energy  in  the  United  States  are 
complex  activities  that  are  slow  to  respond  to  change.     Thus,  the 
investments  already  made  for  installed  electrical  generating  capacity 
and  the  more  than  100  million  vehicles  currently  in  use  have  committed 
us,  over  the  short-term,   to  established  patterns  of  energy  consumption. 
Our  only  option  in  this  time  frame  is  to  continue  to  supply  adequate 
amounts  of  those  fuels  that  now  constitute  our  energy  mix. 

Not  all  of  the  short-term  potential  problems  lend  themselves  to 
solution  through  advanced  technology.     For  example,   a  shortage  in  gase- 
ous fuels  is  expected  to  be  in  full  force  within  the  next  few  years 
despite  anything  that  might  be  done.     This  shortage  is  already  having 
an  effect  on  applications  for  new  customers;  some  industrial  and  other 
users  will  be  forced  to  use  alternate  energy  sources- -petroleum  and 
coal.     Adequate  domestic  supplies  of  these,  commodities  cannot  be  pro- 
duced quickly  and/or  used  freely  under  current  environmental  standards, 
forcing  increasing  reliance  on  low-sulfur  imports  of  foreign  oil. 

Although  improved  technology  will  not  solve  all  short-term  energy 
problems,    it  can  contribute  to  substantial  reductions  in  potential 
impacts  for  many  of  them.     Over  the  longer  term,   new  and  revolutionary 
technology  may  be  developed  and  adopted  that  will  help  provide  energy 
in  the  forms  and  amounts  needed  by  the  Nation  while  at  the  same  time 
preventing  environmental  degradation.     Some  of  the  more  important 

\_l     Needed  technologic  developments  are  actually  more  complex 

than  as  indicated  in  this  section.     For  a  more  thorough  dis- 
cussion see:     U.S.  Department  of  the  Interior,   United  States 
■  Energy;  A  S;;mnijry  Review.   January  1972,   and  Energy  Research 
Needs .   a  report  to  the-  National  Science  Foundation  by  Resources 
for  the  Future,   Inc.,    in  cooperation  with  MIT  Environmental 
Laboratory,   October  1971, 


B-10 


possible  approaches  that  are  expected  to  yield  results  over  the  short, 
intermediate,   and  long  term  are  discussed  in  the  sections  that  follow. 

Short  term  needs   (1972-1980)  -  An  immediate  problem  is  to  increase 
our  available  energy  supply  base  and  develop  means  by  which  these 
resources  can  be  used  without  environmental  degradation.     One  way  to 
increase  supplies  is  by  a  concentrated  effort  to  stimulate  production 
from  known  oil  and  gas  fields.     Such  fields  contain  nearly  60  billion 
barrels  of  oil  and  300  trillion  cubic  feet  of  gas  that  do  not  lend 
themselves  to  economical  recovery  at  current  prices  with  existing 
technology.     Methods  that  may  be  employed  in  an  effort  to  recover  these 
resources  include 

o    fracturing  methods:     nuclear,   chemical  explosives,  hydraulic, 
and  gas ; 

o    displacement  techniques:     improved  water  and  gas  flooding, 

miscible,    solvent,   and  thermal  drives; 
o    production  methods:     fracture  orientation  and  subsurface  hydraulic 

fracture  mapping,  and 
o    combinations  of  various  methods  that  would  improve  total  recovery. 
Increased  exploration  and  development  for  all  of  the  Nation's 
primary  energy  resources  must  be  stimulated,   including  the  search  for 
new,   low-sulfur  coal  deposits  and  low-cost  uranium.     Developments  are. 
needed  in 

o    exploration  techniques:     radiometric,   seismic,  magnetic,  radia- 
tion halo,    infrared,   ultraviolet  and  laser  photography,  and 
side  blocking  radar  and  side  looking  sonar; 

o     production  and  distribution  techniques  for  new  production  areas: 
Northern  Alaska  and  offshore;  and 

o    means  to  convert  coal  to  gaseous  utility  fuels. 

As  supplies  and  consumption  of  these  energy  fuels  increase,  methods 
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must  also  be  developed  to  use  them  in  a  manner  designed  to  enhance  the 
environment.     Of  importance  are: 
-       o    methods  to  reduce  sulfur  discharges  from  stack  gases;  and 

o    fuel/engine  changes  to  restrict  undesirable  vehicular  emissions 
with  minimal   impact  on  combustion  efficiency. 
•    Intermediate  term  needs  (1975-1985)  -  This  period  represents  the 
transitional  time  between,   and  thus  overlaps^   the  short  and  long-run 
time  frame.     Of  importance  in  this  period  is  the  development  of  indigen- 
ous energy  resources  in  the  form  most  needed,  including 

:.  o     conversion  of  coal  to  liquid  fuels  and  pipeline  quality  gaseous 
fuels; 

o    production  of  gaseous  and  liquid  fuels  from  organic  wastes;  and 
o     production  of  liquid  and  gaseous  products  from  the  Nation's 
.    oil  shale,  heavy  oil,   and  tar  sand  deposits,   and  methods  of 

processing  them.   *   .  , 

A  distinguishing  feature  of  each  of  the  above  is  that  the  primary 
resource  is  a  solid  or  semi-solid.     Except  for  organic  wastes,  each 
must  be  processed  in  situ  or  mined,    then  it  must  be  converted  to  a 
usable  liquid  or  gaseous  form.     The  materials  extraction  and  handling 
problems  will  be  on  a  scale  never  before  undertaken  and  current  tech- 
nology is  not  adequate  for  the  task.     Significant  developments  are 
needed  in  ' 

o    materials  handling  systems  to  process  bulk  materials  at  an 

unprecedented  rate; 
o    developing  new  concepts  that  would  optimize  resource  recovery, 
provide  for  underground  placement  of  waste  products  for  local 
support  and  subsidence  control,   and  protect  the  health  and 

safety  of  personnel  and  the  environment; 
o     in  situ  techniques  to  recover  energy  fuels;  and 
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o    training  greater  numbers  of  engineers  and  skilled  workers  in 
the  general  minerals  technology  field. 

Long  term  needs  (1980-2000)  -  The  Nation  is  rapidly  moving  toward 
electricity  as  the  predominate  energy  form.     But  on  the  average^  less 
than  a  third  of  the  energy  going  into  che  boiler  of  a  thermal  electric 
powerplant  emerges  at  the  bus  bar  in  the  form. of  available  electrical 
output.     Nuclear  plants  now  being  built  approximate  the  general  nation 
wide  average  of  32  percent  power  generating  efficiency,  while  the  very 
best  steam  electric  plants  attain  an  efficiency  level  of  only  40  per- 
cent.    Moreover,   from  5  to  10  percent  of  the  generated  electricity  is 
lost  during  transmission.     Every  effort  must  be  made  to  increase  the 
conversion  efficiency  of  our  power  generating  and  transmission  methods 
Chief  options  that  need  to  be  pursued  include 

o     improving  the  methods  of  upgrading  fossil  fuels; 

o    developing  advanced  power  cycles; 

o    improving  fossil  fuels  conversion  processes:     fuel  cells^  MHD, 

thermionic,   and  thermoelectric; 
o    developing  more  efficient  nuclear  breeder  reactors; 
o    developing  new  energy  systems:     nuclear  fusion,   tidal,  wind, 

and  solar;  and 

o    developing  unique  energy  transmission  systems:     electricity  at 
cryogenic  temperatures  and  natural  gas  and  hydrogen  moving  in 
liquefied  form  through  pipelines. 
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'    .  ENERGY  RESEARCH  PROJECT  SELECTION 

The  preceding  section  outlined  some  of  the  research  needed,   if  we 
are  to  meet  the  Nation's  future  demands  for  clean  energy.  Energy 
Research  can  continue  to  make  many  important  contributions.  However, 
certain  boundaries  on  our  activities  and  establishment  of  priorities 
are  required,   if  our  expertise  is  to  be  effectively  utilized. 

To  establish  a  conceptual  framework  for  our  role,   research  might 
be  viewed  as  a  continuous  flow  of  information  that  begins  with  basic 
discoveries  and  continues  through  corriniercial  development.     Such  a  flow 


can  be  illustrated  as 


Within  this  framework,  our  research  covers  some  of  the  basic  area, 
the  applied  area,   and  some  of  the  engineering  area,   all  of  which  fall 
within  the  boundaries  established  by  our  mission  concept.     Full  develop- 
ment of  any  discovery  to  commercial  practice  is  rarely  pursued,   as  is 
basic  research  which  seeks  "knowledge  for  the  sake  of  knowledge  alone." 


The  problem-oriented  research  we  conduct  is  both  useful  and  essential 
for  the  development  of  new  processes  and  products  needed  by  our  society, 
whether  for  economic,   cultural,  or  military  applications.  Ordinarily, 
industry  supports  only  that  research  which  has  a  practical  aspect-- 
that  which  leads  to  increased  profit  margins  and  new  corporate  earn- 
ings.    Moreover,   this  is  usually  on  a  short  range  payout  basis,  generally 
less  than  5  years.     General  guidelines  to  determine  whether  a  particular 
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research  effort  is  suitable  for  action  by  the  Federal  Government  are 
detailed  in  Appendix  B. 

If  a  particular  activity  is  judged  to  be  suitable  for  research 
by  the  Federal  Government^   the  proposed  research  will  be  considered 
appropriate  for  Energy  Research,   provided  it  also  receives  affirmative 
judgments  in  three  major  regards:     (1)  The  research  represents  progress 
toward  accomplishing  our  assigned  mission,    (2)  the  research  is  unique 
or  complementary  in  reference  to  prior,   ongoing,   or  anticipated  research 
by  others,   and  (3)  adequate  staff  and  other  resources  can  be  provided 
to  accomplish  the  research  in  a  timely  fashion.     If  these  criteria 
are  met  affirmatively,   detailed  proposals  will  be  developed  for  con- 
sideration in  our  proposed  budget.     Priorities  for  the  proposed  work 
will  be  established  by  the  Assistant  Direc tor- -Energy  in  consultation 
with  the  appropriate  Division  Chief  and  Research  Director. 

Ordering  of  priorities  requires  and  will  receive  continual  review 
as  the  results  of  ongoing  programs  are  evaluated  and  as  new  ideas  prove 
worthy  of  increased  funding  support.     Our  best  assessment  of  current 
program  emphasis  is  contained  in  the  following  section;   this  list  will 
be  revised  as  needed  to  guide  our  overall  efforts  toward  the  attainment 
of  our  mission. 
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